Bot Directory

Search the Botcrawl Bot Directory database to identify AI crawlers, search bots, scrapers, scanners, monitoring bots, preview bots, and user-agent patterns.

660 bots
Splunk Synthetics Safe

Splunk Synthetics

Synthetic monitoring tool with global agents running in AWS.

Operator and family: Splunk Type: Monitoring
Verified robots.txt: No Block: No Verified Bot
SpyglassesBot Neutral

Spyglasses Bot

Spyglasses Bot evaluates website content to identify issues that would affect visibility by AI assistants.

Operator and family: Spyglasses Type: Ai Purpose: Ai Assistant
Verified robots.txt: No Block: Depends Verified Bot
stably Safe

Stably

Runs functionality tests and user-flow regression checks on websites.

Operator and family: Stably Type: Monitoring
Verified robots.txt: Unknown Block: Depends Verified Bot
Stape Safe

Stape Scanner

Stape Scanner monitoring configuration of tracking tags.

Operator and family: Stape Inc Type: Monitoring
Verified robots.txt: No Block: No Verified Bot
Statabot Neutral

Statabot

Statabot searches for stata.toc files and indexes their contents.

Operator and family: StataCorp LLC Type: Search Purpose: Indexing
Verified robots.txt: No Block: Depends Verified Bot
StatistikAustria Neutral

StatistikAustria

Bot to collect product prices for the official consumer price index of Austria.

Operator and family: Statistik Bot Type: Search Purpose: Indexing
Verified robots.txt: No Block: Depends Verified Bot
StatsDroneBot Neutral

StatsDroneBot

The StatsDrone affiliate marketing statistics scraping and aggregating tool.

Operator and family: StatsDrone Type: Search Purpose: Indexing
Verified robots.txt: No Block: Depends Verified Bot
statuscake Safe

StatusCake

The StatusCake uptime monitor is used to monitor webpage availability and performance.

Operator and family: StatusCake Type: Monitoring
Verified robots.txt: No Block: No Verified Bot
SteamChat Neutral

Steam Chat

The Steam Chat bot fetches previews of URLs shared within the Steam client's chat feature.

Operator and family: Valve Software Type: Scraper Purpose: Scraping
Verified robots.txt: No Block: Depends Verified Bot
Stripe/1.0 Safe

Stripe

The Stripe Webhooks service allows Stripe to push real-time event data to customers' application webhook endpoint when events happen in their Stripe account.

Operator and family: Stripe Type: Webhook
Verified robots.txt: No Block: No Verified Bot
Stripebot Safe

Stripebot

Stripebot is Stripe’s automated crawler for retrieving website data used to support Stripe services and financial compliance workflows.

Operator and family: Stripe Type: Ecommerce
Verified robots.txt: No Block: No Verified Bot
Sucuri Safe

Sucuri

The Sucuri bot is part of the Sucuri website security platform. It crawls websites to scan for malware, security risks, and blacklisting status.

Operator and family: Sucuri Type: Security
Verified robots.txt: No Block: No Verified Bot
sender-9YMgn Safe

Svix Webhooks

Scalable webhook platform featuring automatic retries, signature verification, deep observability, and a static-IP delivery bot—deploy hosted or self-hosted.

Operator and family: Svix Inc. Type: Webhook
Verified robots.txt: No Block: No Verified Bot
SwifteqLinkChecker Safe

Swifteq Link Checker

Swifteq Link Checker is a monitoring bot from Swfiteq Ltd used to check website availability, performance, or page health.

Operator and family: Swfiteq Ltd Type: Monitoring
Verified robots.txt: No Block: No Verified Bot
Datadog Safe

Synthetic Bot

Datadog Synthetics gives you a new layer of visibility on the Datadog platform. By monitoring your applications and API endpoints via simulated user requests and browser rendering.

Operator and family: DataDog Type: Monitoring
Verified robots.txt: No Block: No Verified Bot
Taboolabot Neutral

Taboola

The Taboola crawler visits websites advertiser campaign websites to audit the content of the page and gather site metadata and summary data.

Operator and family: Taboola Type: Search Purpose: Indexing
Verified robots.txt: No Block: Depends Verified Bot
TelegramBot Neutral

Telegram Bot

TelegramBot crawls websites to render a link preview when people send a message containing a URL in the Telegram messaging service.

Operator and family: Telegram Type: Preview
Not verified robots.txt: No Block: Depends Observed
TermlyBot Neutral

Termly

Termly is a security bot associated with Termly, used for security, verification, or anti-abuse checks.

Operator and family: Termly Type: Security
Verified robots.txt: No Block: Depends Verified Bot
Terracotta Safe

Terracotta

The Terracotta bot scrapes websites for use in generating indices for serving searches using Ceramic's search product.

Operator: Ceramic Family: Terracotta Type: Scraper Purpose: Scraping
Verified robots.txt: No Block: Depends Verified Bot
curl/7.58.0-DEV Safe

Thousand Eyes Cloud Agent

ThousandEyes monitors network infrastructure, troubleshoots application delivery and maps Internet performance, all from a SaaS-based platform.

Operator and family: Thousand Eyes Type: Monitoring
Verified robots.txt: No Block: No Verified Bot
TikTok Safe

TikTok

TikTok is a preview or prefetch bot from ByteDance used to fetch pages for previews, unfurls, or user-triggered rendering.

Operator and family: ByteDance Type: Preview
Verified robots.txt: Yes Block: Depends Verified Bot
TikTokSpider Safe

TikTokSpider

TikTokSpider is a crawler used for scraping, SEO analysis, or data collection from ByteDance.

Operator and family: ByteDance Type: Scraper Purpose: Scraping
Verified robots.txt: Yes Block: Depends Verified Bot
Timpibot Neutral

Timpibot

Timpi decentralized crawler observed in logs for web indexing and AI-oriented data collection.

Operator and family: Timpi Type: Ai Purpose: AI Data Collection
Not verified robots.txt: No Block: Depends Observed
Bytespider Safe

Toutiao

Toutiao is a search crawler from ByteDance used for search indexing, page discovery, or content rendering.

Operator and family: ByteDance Type: Search Purpose: Indexing
Verified robots.txt: Yes Block: Depends Verified Bot