About this bot
CitibotSiteCrawler is a crawler associated with Citibot and used for civic content ingestion.
It can be identified in HTTP traffic by the CitibotSiteCrawler identity family, including CitibotSiteCrawler/1.0.
Directory status: Active. Identity classification: Verified with Medium confidence. Identity type: documented.
robots.txt behavior is not sufficiently established to claim compliance.
Match CitibotSiteCrawler/1.0 and corroborate it with the expected civic-site ingestion pattern.
How to identify this bot
- User-agent and crawler identifiers
-
CitibotSiteCrawlerCitibotSiteCrawler/1.0
Details
- Kind
- Crawler
- Family
- CitibotSiteCrawler
- Purpose
- Civic content ingestion
- Common Use
- Retrieving public government and civic website content for Citibot civic-engagement and AI knowledge services.
- Respects robots.txt
- Unknown
Verification
- Identity Type
- Documented
- Evidence Method
- Match CitibotSiteCrawler/1.0 and corroborate it with the expected civic-site ingestion pattern.
- Detection Notes
- The verified purpose is civic/public-content ingestion; do not label the traffic as model training without separate evidence.
Website controls
- Robots.txt Snippet
User-agent: CitibotSiteCrawler Disallow: /