Bot Directory

Search public Botcrawl bot records to identify AI crawlers, search bots, scrapers, scanners, monitoring bots, preview bots, and user-agent patterns.

AI Crawlers Search Bots Scrapers Scanners Monitoring Preview Bots
See these bots hitting your site in real time

The public directory explains known bots and user-agents. Botcrawl Edge uses BotDB plus datacenter intelligence for live classification, rules, and enforcement.

Try Edge Free Open Edge BotDB
998 bots
J-BRW Neutral

Yahoo Japan SEO Crawler

Yahoo Japan SEO Crawler is a search crawler from Yahoo used for search indexing, URL discovery, page rendering, result freshness.

Operator and family: Yahoo Type: Search Purpose: Indexing
Verified robots.txt: No Block: Depends Verified Bot
View Record Open in Edge BotDB
Y!J-ASR Neutral

Yahoo Japan Y!J-ASR

Yahoo Japan Y!J-ASR is a search crawler from Yahoo Japan Corporation used for search indexing, URL discovery, page rendering, result freshness.

Operator and family: Yahoo Japan Corporation Type: Search Purpose: Indexing
Verified robots.txt: Unknown Block: Depends Verified Bot
View Record Open in Edge BotDB
Yahoo Link Preview Neutral

Yahoo Link Preview

Yahoo Link Preview is a link preview bot from Yahoo used for link unfurling, rich-card generation, social previews, messaging previews.

Operator and family: Yahoo Type: Preview
Verified robots.txt: No Block: Depends Verified Bot
View Record Open in Edge BotDB
Yahoo-MMCrawler Neutral

Yahoo MMCrawler

Yahoo MMCrawler is a search crawler from Yahoo used for search indexing, URL discovery, page rendering, result freshness.

Operator and family: Yahoo Type: Search Engine Crawler Purpose: Media Indexing
Not verified robots.txt: Unknown Block: Depends Documented
View Record Open in Edge BotDB
Yahoo! Slurp Safe

Yahoo Slurp

Yahoo Slurp is a search crawler from Yahoo used for search indexing, URL discovery, page rendering, result freshness.

Operator and family: Yahoo Type: Search Purpose: Indexing
Verified robots.txt: Yes Block: Depends Verified Bot
View Record Open in Edge BotDB
J-DLC Safe

Yahoo! JAPAN J-DLC

Yahoo! JAPAN J-DLC is a search crawler from Yahoo! JAPAN used for search indexing, URL discovery, page rendering, result freshness.

Operator and family: Yahoo! JAPAN Type: Search Purpose: Indexing
Verified robots.txt: Yes Block: Depends Verified Bot
View Record Open in Edge BotDB
YahooCacheSystem Neutral

YahooCacheSystem

YahooCacheSystem is a search crawler from Yahoo used for search indexing, URL discovery, page rendering, result freshness.

Operator and family: Yahoo Type: Search Purpose: Indexing
Verified robots.txt: Unknown Block: Depends Verified Bot
View Record Open in Edge BotDB
YahooMailProxy Neutral

YahooMailProxy

YahooMailProxy is a security scanner from Yahoo used for security scanning, malware checks, vulnerability assessment, certificate review.

Operator and family: Yahoo Type: Security
Verified robots.txt: No Block: Depends Verified Bot
View Record Open in Edge BotDB
YahooSeeker Neutral

YahooSeeker

YahooSeeker is a search crawler from Yahoo used for search indexing, URL discovery, page rendering, result freshness.

Operator and family: Yahoo Type: Search Engine Crawler Purpose: Search Indexing
Not verified robots.txt: Unknown Block: Depends Documented
View Record Open in Edge BotDB
YaK Neutral

YaK

YaK is an AI crawler used for AI crawler discovery, assistant retrieval, AI search support, or model-related web access.

Operator and family: YaK Type: Ai Purpose: Ai Crawl
Not verified robots.txt: Unknown Block: Depends Observed
View Record Open in Edge BotDB
YandexAccessibilityBot Safe

YandexAccessibilityBot

YandexAccessibilityBot is a search crawler from Yandex used for search indexing, URL discovery, page rendering, result freshness.

Operator and family: Yandex Type: Search Purpose: Indexing
Verified robots.txt: Yes Block: Depends Official Documented
View Record Open in Edge BotDB
YandexAdditional Neutral

YandexAdditional

YandexAdditional is a search crawler from Yandex used for search indexing, URL discovery, page rendering, result freshness.

Operator and family: Yandex Type: Search Purpose: Indexing
Verified robots.txt: Yes Block: Depends Official Documented
View Record Open in Edge BotDB
YandexAdditionalBot Neutral

YandexAdditionalBot

YandexAdditionalBot is a search crawler from Yandex used for search indexing, URL discovery, page rendering, result freshness.

Operator and family: Yandex Type: Search Purpose: Indexing
Verified robots.txt: Yes Block: Depends Official Documented
View Record Open in Edge BotDB
YandexAdNet Safe

YandexAdNet

YandexAdNet is a search crawler from Yandex used for search indexing, URL discovery, page rendering, result freshness.

Operator and family: Yandex Type: Search Purpose: Indexing
Verified robots.txt: Yes Block: Depends Official Documented
View Record Open in Edge BotDB
YandexBlogs Safe

YandexBlogs

YandexBlogs is a search crawler from Yandex used for search indexing, URL discovery, page rendering, result freshness.

Operator and family: Yandex Type: Search Purpose: Indexing
Verified robots.txt: Yes Block: Depends Official Documented
View Record Open in Edge BotDB
YandexBot Safe

YandexBot

YandexBot is a search crawler from Yandex used for search indexing, URL discovery, page rendering, result freshness.

Operator and family: Yandex Type: Search Purpose: Indexing
Verified robots.txt: Yes Block: Depends Official Documented
View Record Open in Edge BotDB
YandexCalendar Safe

YandexCalendar

YandexCalendar is a search crawler from Yandex used for search indexing, URL discovery, page rendering, result freshness.

Operator and family: Yandex Type: Search
Verified robots.txt: Yes Block: Depends Official Documented
View Record Open in Edge BotDB
YandexComBot Safe

YandexComBot

YandexComBot is a search crawler from Yandex used for search indexing, URL discovery, page rendering, result freshness.

Operator and family: Yandex Type: Search Purpose: Indexing
Verified robots.txt: Yes Block: Depends Official Documented
View Record Open in Edge BotDB
YandexDirect Safe

YandexDirect

YandexDirect is a search crawler from Yandex used for search indexing, URL discovery, page rendering, result freshness.

Operator and family: Yandex Type: Search Purpose: Indexing
Verified robots.txt: Yes Block: Depends Official Documented
View Record Open in Edge BotDB
YandexDirectDyn Safe

YandexDirectDyn

YandexDirectDyn is a search crawler from Yandex used for search indexing, URL discovery, page rendering, result freshness.

Operator and family: Yandex Type: Search Purpose: Indexing
Verified robots.txt: Yes Block: Depends Official Documented
View Record Open in Edge BotDB
YandexFavicons Safe

YandexFavicons

YandexFavicons is a link preview bot from Yandex used for link unfurling, rich-card generation, social previews, messaging previews.

Operator and family: Yandex Type: Preview
Verified robots.txt: Yes Block: Depends Official Documented
View Record Open in Edge BotDB
YandexImageResizer Safe

YandexImageResizer

YandexImageResizer is a search crawler from Yandex used for search indexing, URL discovery, page rendering, result freshness.

Operator and family: Yandex Type: Search Purpose: Indexing
Verified robots.txt: Yes Block: Depends Official Documented
View Record Open in Edge BotDB
YandexImages Safe

YandexImages

YandexImages is a search crawler from Yandex used for search indexing, URL discovery, page rendering, result freshness.

Operator and family: Yandex Type: Search Purpose: Indexing
Verified robots.txt: Yes Block: Depends Official Documented
View Record Open in Edge BotDB
YandexMarket Safe

YandexMarket

YandexMarket is a search crawler from Yandex used for search indexing, URL discovery, page rendering, result freshness.

Operator and family: Yandex Type: Search Purpose: Indexing
Verified robots.txt: Yes Block: Depends Official Documented
View Record Open in Edge BotDB