Bot Directory

Search the Botcrawl Bot Directory database to identify AI crawlers, search bots, scrapers, scanners, monitoring bots, preview bots, and user-agent patterns.

660 bots
WPMU DEV Broken Link Checker Safe

WPMU DEV Broken Link Checker

WPMU DEV Broken Link Checker is a monitoring bot from WPMUDEV used to check website availability, performance, or page health.

Operator and family: WPMUDEV Type: Monitoring
Verified robots.txt: No Block: No Verified Bot
WPMUDEV Safe

WPMUDEV Uptime Monitor 5.0

WPMUDEV Uptime Monitor 5.0 is a monitoring bot from WPMUDEV used to check website availability, performance, or page health.

Operator and family: WPMUDEV Type: Monitoring
Verified robots.txt: No Block: No Verified Bot
xAI-Grok Neutral

xAI-Grok

Claimed xAI Grok user-agent token for Grok-related web access.

Operator: xAI Family: xAI Grok Type: Ai Purpose: AI Search Or Agentic Browsing
Not verified robots.txt: Unknown Block: Depends Observed
XY-Archive-Compliance Neutral

XY Archive Compliance Bot

XY Archive Compliance Bot is a security scanner from XY Archive Compliance used for security checks, malware review, or abuse prevention.

Operator and family: XY Archive Compliance Type: Security
Verified robots.txt: No Block: Depends Verified Bot
yacybot Neutral

yacybot

yacybot is a search crawler from YaCy used for search indexing, page discovery, or content rendering.

Operator and family: YaCy Type: Search Purpose: Indexing
Verified robots.txt: Yes Block: Depends Official Documented
YaDirectFetcher Safe

YaDirectFetcher

YaDirectFetcher is a monitoring bot from Yandex used to check website availability, performance, or page health.

Operator and family: Yandex Type: Monitoring
Verified robots.txt: No Block: Depends Official Documented
Yahoo Ad monitoring Neutral

Yahoo Ad Monitoring

Yahoo Ad Monitor monitors the contents of webpages where Yahoo! ads are served.

Operator and family: Yahoo Type: Monitoring
Verified robots.txt: No Block: Depends Verified Bot
J-BRW Neutral

Yahoo Japan SEO Crawler

Yahoo Japan search engine crawler for SEO analysis.

Operator and family: Yahoo Type: Search Purpose: Indexing
Verified robots.txt: No Block: Depends Verified Bot
Y!J-ASR Neutral

Yahoo Japan Y!J-ASR

Yahoo Japan Y!J-ASR is a search crawler associated with Yahoo Japan Corporation, used for search indexing or content discovery.

Operator and family: Yahoo Japan Corporation Type: Search Purpose: Indexing
Verified robots.txt: No Block: Depends Verified Bot
Yahoo Link Preview Neutral

Yahoo Link Preview

Yahoo Link Preview's bot fetches data from URLs shared on Yahoo platforms.

Operator and family: Yahoo Type: Preview
Verified robots.txt: No Block: Depends Verified Bot
Yahoo! Slurp Safe

Yahoo Slurp

Yahoo! Slurp was the search engine crawler for Yahoo's search engine.

Operator and family: Yahoo Type: Search Purpose: Indexing
Verified robots.txt: Yes Block: Depends Verified Bot
J-DLC Safe

Yahoo! JAPAN J-DLC

Yahoo! JAPAN manages and operates a system that accesses web pages published on the Internet for the purpose of providing services, research, development, maintenance, etc.

Operator and family: Yahoo! JAPAN Type: Search Purpose: Indexing
Verified robots.txt: Yes Block: Depends Verified Bot
YahooCacheSystem Neutral

YahooCacheSystem

YahooCacheSystem caches website contents as part of the Yahoo! Search Service.

Operator and family: Yahoo Type: Search Purpose: Indexing
Verified robots.txt: No Block: Depends Verified Bot
YahooMailProxy Neutral

YahooMailProxy

Yahoo Mail Proxy is a content fetch proxy that retrieves the page content of URLs that are embedded within emails sent to Yahoo Mail users. Having the content displayed through the proxy.

Operator and family: Yahoo Type: Security
Verified robots.txt: No Block: Depends Verified Bot
YandexAccessibilityBot Safe

YandexAccessibilityBot

YandexAccessibilityBot is a search crawler from Yandex used for search indexing, page discovery, or content rendering.

Operator and family: Yandex Type: Search Purpose: Indexing
Verified robots.txt: No Block: Depends Official Documented
YandexAdditional Neutral

YandexAdditional

YandexAdditional is a search crawler from Yandex used for search indexing, page discovery, or content rendering.

Operator and family: Yandex Type: Search Purpose: Indexing
Verified robots.txt: No Block: Depends Official Documented
YandexAdditionalBot Neutral

YandexAdditionalBot

YandexAdditionalBot is a search crawler from Yandex used for search indexing, page discovery, or content rendering.

Operator and family: Yandex Type: Search Purpose: Indexing
Verified robots.txt: No Block: Depends Official Documented
YandexAdNet Safe

YandexAdNet

YandexAdNet is a search crawler from Yandex used for search indexing, page discovery, or content rendering.

Operator and family: Yandex Type: Search Purpose: Indexing
Verified robots.txt: Yes Block: Depends Official Documented
YandexBlogs Safe

YandexBlogs

YandexBlogs is a search crawler from Yandex used for search indexing, page discovery, or content rendering.

Operator and family: Yandex Type: Search Purpose: Indexing
Verified robots.txt: Yes Block: Depends Official Documented
YandexBot Safe

YandexBot

YandexBot is a search crawler from Yandex used for search indexing, page discovery, or content rendering.

Operator and family: Yandex Type: Search Purpose: Indexing
Verified robots.txt: Yes Block: Depends Official Documented
YandexCalendar Safe

YandexCalendar

YandexCalendar is a search crawler from Yandex used for search indexing, page discovery, or content rendering.

Operator and family: Yandex Type: Search
Verified robots.txt: No Block: Depends Official Documented
YandexComBot Safe

YandexComBot

YandexComBot is a search crawler from Yandex used for search indexing, page discovery, or content rendering.

Operator and family: Yandex Type: Search Purpose: Indexing
Verified robots.txt: No Block: Depends Official Documented
YandexDirect Safe

YandexDirect

YandexDirect is a search crawler from Yandex used for search indexing, page discovery, or content rendering.

Operator and family: Yandex Type: Search Purpose: Indexing
Verified robots.txt: No Block: Depends Official Documented
YandexDirectDyn Safe

YandexDirectDyn

YandexDirectDyn is a search crawler from Yandex used for search indexing, page discovery, or content rendering.

Operator and family: Yandex Type: Search Purpose: Indexing
Verified robots.txt: No Block: Depends Official Documented