Crawlspace
Crawlspace is an automated agent from Crawlspace used for user configured web crawling.
Daric2
Daric2 is a cryptographically signed Cloudflare AI Crawl Control identity. It does not publish a fixed HTTP User-Agent.
Daric3
Daric3 is a cryptographically signed Cloudflare AI Crawl Control identity. It does not publish a fixed HTTP User-Agent.
Daric4
Daric4 is a cryptographically signed Cloudflare AI Crawl Control identity. It does not publish a fixed HTTP User-Agent.
DeepSeekBot
DeepSeekBot remains an observed, Unverified DeepSeek-attributed crawler token; public web-data use is documented, but the exact crawler identity is not.
DesearchBot
DesearchBot is a Desearch crawler used for indexing public pages for search and retrieval by people and AI agents.
Devin
Devin is Cognition's autonomous software-engineering agent with browser capabilities, but the literal `Devin` User-Agent remains Unverified.
DigitalOcean Gradient AI Crawler
DigitalOceanGradientAICrawler/1.0 indexes websites configured as DigitalOcean knowledge-base sources and follows robots.txt.
Doubaobot
Doubaobot is an observed ByteDance/Doubao-attributed crawler token. Its exact purpose and authentication model remain insufficiently documented.
ExaSearchBot
ExaSearchBot is a Exa crawler used for search indexing and retrieval for Exa search products.
FishBot
FishBot is a verified crawler associated with Fish Audio used for ai crawling. It is commonly identified by `FishBot`.
FriendlyCrawler
FriendlyCrawler is an observed crawler attributed to Unknown used for ai crawling. It is commonly identified by `FriendlyCrawler`.
GeistHaus PageFetcher
GeistHaus PageFetcher is an observed fetcher attributed to GeistHaus used for ai assistant page fetch. It is commonly identified by `GeistHaus-PageFetcher`.
Gemini Deep Research
Gemini Deep Research is an observed fetcher attributed to Google used for user triggered ai research. It is commonly identified by `gemini-deep-research`.
Gist Crawler
The Gist Crawler retrieves licensed and public web content for Gist.ai search and retrieval.
Google Gemini CLI
Google Gemini CLI is a verified software associated with Google / Gemini CLI users used for ai assistant. It is commonly identified by…
Google-Agent
Google-Agent is associated with Google and is used for navigating and acting on websites in response to users of agents hosted on Google…
Google-CloudVertexBot
Google-CloudVertexBot is associated with Google and is used for site-owner-requested crawling used to build or refresh Vertex AI Agent data; it is not…
Google-GeminiNotebook
Google-GeminiNotebook is associated with Google and is used for fetching sources explicitly supplied by users to Gemini Notebook / NotebookLM experiences. It is…