Bot Directory

Search the Botcrawl Bot Directory database to identify AI crawlers, search bots, scrapers, scanners, monitoring bots, preview bots, and user-agent patterns.

660 bots
Chrome-Lighthouse Neutral

Chrome-Lighthouse

Chrome-Lighthouse is an automated, open-source tool for auditing web page quality and does not operate as a traditional web crawler. It runs a series of audits against a given page to.

Operator and family: Google Type: Search Purpose: Indexing
Verified robots.txt: No Block: Depends Verified Bot
CitibotSiteCrawler Neutral

CitibotSiteCrawler

CitibotSiteCrawler collects public data from government websites to power Citibot’s AI civic engagement tools.

Operator: Citibot Family: CitibotSiteCrawler Type: Search Purpose: Indexing
Verified robots.txt: No Block: Depends Verified Bot
Claude-Code Neutral

Claude-Code

Claimed Claude Code user-agent identifier associated with Anthropic developer-agent workflows.

Operator: Anthropic Family: Claude Type: Ai Purpose: Developer Agent Fetch
Not verified robots.txt: Unknown Block: Depends Observed
Claude-SearchBot Safe

Claude-SearchBot

Claude-SearchBot is Anthropic’s search-focused crawler for retrieving web content.

Operator and family: Anthropic Type: Ai Purpose: Ai Search
Verified robots.txt: Yes Block: Depends Official Documented
Claude-User Neutral

Claude-User

Claude-User represents user-initiated Claude retrieval activity rather than broad background crawling.

Operator and family: Anthropic Type: Ai Purpose: Ai Assistant
Verified robots.txt: Unknown Block: Depends Official Documented
ClaudeBot Neutral

ClaudeBot

ClaudeBot is Anthropic’s crawler used to discover and retrieve public web content for Claude-related systems.

Operator and family: Anthropic Type: Ai Purpose: Ai Crawl
Verified robots.txt: Yes Block: Depends Official Documented
Clearscopebot Neutral

ClearscopeBot

Clearscope is an AI-driven SEO content optimization platform developed by Mushi Labs. It assists content creators, marketers, and SEO professionals in producing high-quality.

Operator and family: Mushi Labs Type: Search Purpose: Indexing
Verified robots.txt: Yes Block: Depends Verified Bot
CledaraBot Neutral

Cledara SaaS Management Agent

Cledara SaaS Management Agent is a commerce-related fetcher from Cledara used for commerce, checkout, advertising, or payment-related checks.

Operator and family: Cledara Type: Ecommerce
Verified robots.txt: No Block: Depends Verified Bot
Clickagy Intelligence Bot v2 Neutral

Clickagy

The Clickagy Intelligence Bot is an ad verification bot for Clickagy.

Operator and family: Clickagy Type: Security
Verified robots.txt: No Block: Depends Verified Bot
Cloudflare Crawler Safe

Cloudflare Browser Rendering Crawler

Cloudflare crawler used by Browser Rendering crawl workflows and identified in Cloudflare Radar.

Operator and family: Cloudflare Type: Ai Purpose: Browser Rendering Crawl
Verified robots.txt: Yes Block: No Verified Bot
Cloudtrellis Safe

Cloudtrellis

Cloudtrellis automatically scans your entire site for broken links, accessibility issues, and potential SEO improvements.

Operator and family: Cloudtrellis Type: Monitoring
Verified robots.txt: No Block: No Verified Bot
cludo.com Safe

Cludo

Cludobot crawls websites to facilitate and provide search and analytics solutions for its customers.

Operator and family: Cludo Type: Search Purpose: Indexing
Verified robots.txt: Yes Block: No Verified Bot
coccocbot Safe

Cốc Cốc

Coccocbot scrapes websites that are request from the Vietnamese search engine Coc Coc.

Operator and family: Coccoc Type: Scraper Purpose: Scraping
Verified robots.txt: No Block: Depends Verified Bot
coccocbot-fast Safe

coccocbot-fast

coccocbot-fast is a search crawler from Coccoc used for search indexing, page discovery, or content rendering.

Operator and family: Coccoc Type: Search Purpose: Indexing
Verified robots.txt: Unknown Block: Depends Verified Bot
James BOT Neutral

cognitiveSEO Crawler

CognitiveSEO is an SEO toolset that crawls the web and analyzes links.

Operator and family: cognitive SEO Internet Marketing Tools. Type: Search Purpose: Indexing
Verified robots.txt: No Block: Depends Verified Bot
cohere-ai Neutral

cohere-ai

cohere-ai is a search crawler from Cohere used for search indexing, page discovery, or content rendering.

Operator and family: Cohere Type: Search Purpose: Ai Training
Verified robots.txt: Unknown Block: Depends Verified Bot
cohere-training-data-crawler Neutral

cohere-training-data-crawler

cohere-training-data-crawler is a search crawler from Cohere used for search indexing, page discovery, or content rendering.

Operator and family: Cohere Type: Search Purpose: Ai Training
Verified robots.txt: Unknown Block: Depends Verified Bot
weipay-webhooks Safe

Coinbase Webhooks

Coinbase Webhooks are automated messages sent from the Coinbase platform to a user's server, used for notifying users about events such as receiving crypto payments.

Operator and family: Coinbase Type: Webhook
Verified robots.txt: No Block: No Verified Bot
consentcheck Neutral

ConsentCheck Bot

ConsentCheck Bot is a security scanner from ConsentCheck used for security checks, malware review, or abuse prevention.

Operator and family: ConsentCheck Type: Security
Verified robots.txt: No Block: Depends Verified Bot
contentking Safe

ContentKing

ContentKing is a cloud-based service that monitors websites from a digital marketing perspective. We monitor the websites for customers such as Netflix, Atlassian, Fedex and IBM and alert.

Operator and family: ContentKing Type: Monitoring
Verified robots.txt: No Block: No Verified Bot
ContextualBot Neutral

ContextualBot

Accesses page content to create contextual segments for targeting within several DPS & SSP platforms.

Operator and family: Outcomes Type: Scraper Purpose: Scraping
Verified robots.txt: No Block: Depends Verified Bot
Convermax Safe

ConvermaxBot

ConvermaxBot is a search crawler from Convermax Corp. used for search indexing, page discovery, or content rendering.

Operator and family: Convermax Corp. Type: Search Purpose: Indexing
Verified robots.txt: No Block: Depends Verified Bot
CookieHub Safe

Cookie Hub

Cookie Hub is a security scanner from CookieHub used for security checks, malware review, or abuse prevention.

Operator and family: CookieHub Type: Security
Verified robots.txt: No Block: No Verified Bot
cookie-maestro Safe

cookie maestro

cookie maestro is a search crawler from Cookie Maestro used for search indexing, page discovery, or content rendering.

Operator and family: Cookie Maestro Type: Search Purpose: Indexing
Verified robots.txt: Yes Block: No Verified Bot