Bot Directory

Search the public Botcrawl bot directory by name, operator, type, or user-agent.

AI Crawlers Search Bots Scrapers Scanners
Need live enforcement?

Classify traffic and apply rules before content is served.

Try Edge Free Open Edge BotDB
998 bots
RevvimGort Neutral

Revvim

Revvim is a search crawler used for search indexing, URL discovery, page rendering, result freshness.

Operator and family: Revvim Type: Search Purpose: Indexing
Verified robots.txt: No Block: Depends Verified Bot
View Record Open in Edge BotDB
reward-gateway Safe

Reward Gateway

Reward Gateway is a monitoring bot used for security scanning, malware checks, vulnerability assessment, certificate review.

Operator and family: Reward Gateway Type: Security
Verified robots.txt: Unknown Block: No Verified Bot
View Record Open in Edge BotDB
Rival IQ Caution

Rival IQ

Rival IQ is a marketing crawler used for advertising quality review, contextual classification, brand-safety checks.

Operator and family: Rival IQ Type: Marketing Crawler Purpose: Marketing Intelligence
Verified robots.txt: Unknown Block: Monitor Documented
View Record Open in Edge BotDB
Robinbot Neutral

Robinbot

Robinbot is a security scanner from Robin Education used for security scanning, malware checks, vulnerability assessment, certificate review.

Operator: Robin Education Family: Robin Type: Security
Verified robots.txt: No Block: Depends Verified Bot
View Record Open in Edge BotDB
RSSAPI Safe

RSS API

RSS API is a monitoring bot from RSS API (by Tibush GmbH) used for uptime monitoring, synthetic checks, performance measurement, accessibility review.

Operator and family: RSS API (by Tibush GmbH) Type: Monitoring
Verified robots.txt: No Block: Depends Verified Bot
View Record Open in Edge BotDB
rss2tg bot Safe

rss2tg_bot

rss2tg_bot is a feed fetcher from Yellow Rubber Duck Consulting used for RSS or Atom feed polling, syndication refresh, subscription delivery.

Operator and family: Yellow Rubber Duck Consulting Type: Feed Purpose: Feed Fetch
Verified robots.txt: No Block: Depends Verified Bot
View Record Open in Edge BotDB
RSSOwl Neutral

RSSOwl

RSSOwl is a feed fetcher used for RSS or Atom feed polling, syndication refresh, subscription delivery.

Operator and family: RSSOwl Type: Feed Fetcher Purpose: Feed Fetching
Verified robots.txt: Unknown Block: Allow Documented
View Record Open in Edge BotDB
RuxitSynthetic Neutral

RuxitSynthetic

RuxitSynthetic is a monitoring bot from Dynatrace used for uptime monitoring, synthetic checks, performance measurement, accessibility review.

Operator and family: Dynatrace Type: Monitoring Bot Purpose: Uptime And Performance Monitoring
Verified robots.txt: Unknown Block: Allow Documented
View Record Open in Edge BotDB
ryebot Safe

RyeBot

RyeBot is a security scanner from Rye used for security scanning, malware checks, vulnerability assessment, certificate review.

Operator and family: Rye Type: Security
Verified robots.txt: Unknown Block: Depends Verified Bot
View Record Open in Edge BotDB
RyteBot Safe

RyteBot

RyteBot is a search crawler from Semrush used for search indexing, URL discovery, page rendering, result freshness.

Operator and family: Semrush Type: Search Purpose: Indexing
Verified robots.txt: Yes Block: Depends Official Documented
View Record Open in Edge BotDB
SFDC-Callout Neutral

Salesforce

Salesforce is a web scraper from Salseforce used for public web data collection, page extraction, content monitoring.

Operator and family: Salseforce Type: Scraper Purpose: Scraping
Verified robots.txt: Unknown Block: Depends Verified Bot
View Record Open in Edge BotDB
page-preview-tool Neutral

SalesViewerBot

SalesViewerBot is a web scraper from SalesViewer GmbH used for public web data collection, page extraction, content monitoring.

Operator and family: SalesViewer GmbH Type: Scraper Purpose: Scraping
Verified robots.txt: Unknown Block: Depends Verified Bot
View Record Open in Edge BotDB
sanity-webhooks Safe

Sanity Webhooks

Sanity Webhooks is a webhook callback from Sanity used for webhook delivery, payment notifications, service callbacks.

Operator and family: Sanity Type: Webhook
Verified robots.txt: Unknown Block: Depends Verified Bot
View Record Open in Edge BotDB
Sansec Security Monitor Safe

Sansec Security Monitor

Sansec Security Monitor is a security scanner used for security scanning, malware checks, vulnerability assessment, certificate review.

Operator and family: Sansec Security Monitor Type: Security
Verified robots.txt: No Block: No Verified Bot
View Record Open in Edge BotDB
SauceNAO Caution

SauceNAO

SauceNAO is an image Search Crawler used for search indexing, URL discovery, page rendering, result freshness.

Operator and family: SauceNAO Type: Image Search Crawler Purpose: General Bot Traffic
Verified robots.txt: Unknown Block: Monitor Documented
View Record Open in Edge BotDB
SBIntuitionsBot Neutral

SBIntuitionsBot

SBIntuitionsBot is an AI training crawler from SB Intuitions used for AI model training, dataset discovery.

Operator and family: SB Intuitions Type: Ai Purpose: Ai Training
Not verified robots.txt: Unknown Block: Depends Observed
View Record Open in Edge BotDB
ScavoBot Neutral

ScavoBot

ScavoBot is a security scanner from Scavo used for security scanning, malware checks, vulnerability assessment, certificate review.

Operator and family: Scavo Type: Security
Verified robots.txt: Unknown Block: Depends Verified Bot
View Record Open in Edge BotDB
Scoop Caution

Scoop

Scoop is a web scraper from Scoop.it used for public web data collection, page extraction, content monitoring.

Operator and family: Scoop.it Type: Content Crawler Purpose: Content Discovery
Verified robots.txt: Unknown Block: Monitor Documented
View Record Open in Edge BotDB
scour.ing Safe

Scour RSS Bot

Scour RSS Bot is a feed fetcher from Scour used for RSS or Atom feed polling, syndication refresh, subscription delivery.

Operator and family: Scour Type: Feed Purpose: Feed Fetch
Verified robots.txt: No Block: Depends Verified Bot
View Record Open in Edge BotDB
Scrapy Neutral

Scrapy

Scrapy is a web scraper used for public web data collection, page extraction, content monitoring.

Operator and family: Scrapy Type: Scraper Purpose: Web Scraping
Verified robots.txt: Unknown Block: Depends Official Documented
View Record Open in Edge BotDB
Screaming Frog SEO Spider Safe

Screaming Frog SEO Spider

Screaming Frog SEO Spider is a search crawler from Screaming Frog used for search indexing, URL discovery, page rendering, result freshness.

Operator and family: Screaming Frog Type: Search Purpose: Technical SEO Audit
Verified robots.txt: Yes Block: No Official Documented
View Record Open in Edge BotDB
screpy Neutral

Screpy

Screpy is an SEO crawler from screpy.com used for SEO analysis, crawl diagnostics, backlink discovery, technical audit work.

Operator: screpy.com Family: Screpy Type: Seo Crawler Purpose: Technical Seo Auditing
Verified robots.txt: Unknown Block: Depends Documented
View Record Open in Edge BotDB
SE Ranking Bot Safe

SE Ranking Bot

SE Ranking Bot is a search crawler from SE Ranking used for search indexing, URL discovery, page rendering, result freshness.

Operator and family: SE Ranking Type: Search Purpose: Indexing
Not verified robots.txt: Yes Block: Depends Observed
View Record Open in Edge BotDB
SearchAtlas Bot Neutral

SearchAtlas Bot

SearchAtlas Bot is a search crawler from Search Atlas used for search indexing, URL discovery, page rendering, result freshness.

Operator and family: Search Atlas Type: Search Purpose: Indexing
Verified robots.txt: Unknown Block: Depends Verified Bot
View Record Open in Edge BotDB