Bot Directory: Search Bots, Crawlers and User Agents

See these bots hitting your site in real time

The public directory explains known bots and user-agents. Botcrawl Edge uses BotDB plus datacenter intelligence for live classification, rules, and enforcement.

Try Edge Free Open Edge BotDB

998 bots

View by

TermlyBot Neutral

Termly

Termly is a security scanner used for security scanning, malware checks, vulnerability assessment, certificate review.

Operator and family: Termly Type: Security

Verified robots.txt: No Block: Depends Verified Bot

View Record Open in Edge BotDB

Terracotta Safe

Terracotta

Terracotta is a web scraper from Ceramic used for public web data collection, page extraction, content monitoring.

Operator: Ceramic Family: Terracotta Type: Scraper Purpose: Scraping

Verified robots.txt: No Block: Depends Verified Bot

View Record Open in Edge BotDB

TheSkinnyCrawler Neutral

TheSkinnyCrawler

TheSkinnyCrawler is an AI agent from TheSkinnyAI used for assistant-driven browsing, page retrieval.

Operator and family: TheSkinnyAI Type: Ai Assistant Purpose: Site Ai Fetching

Not verified robots.txt: Unknown Block: Depends Documented

View Record Open in Edge BotDB

Thinkbot Neutral

Thinkbot

Thinkbot is an AI crawler used for assistant-driven browsing, page retrieval.

Operator and family: Thinkbot Type: Ai Purpose: Ai Assistant

Not verified robots.txt: Unknown Block: Depends Observed

View Record Open in Edge BotDB

curl/7.58.0-DEV Safe

Thousand Eyes Cloud Agent

Thousand Eyes Cloud Agent is a monitoring bot from Thousand Eyes used for uptime monitoring, synthetic checks, performance measurement, accessibility review.

Operator and family: Thousand Eyes Type: Monitoring

Verified robots.txt: Unknown Block: No Verified Bot

View Record Open in Edge BotDB

TikTok Safe

TikTok

TikTok is a link preview bot from ByteDance used for link unfurling, rich-card generation, social previews, messaging previews.

Operator and family: ByteDance Type: Preview

Verified robots.txt: Yes Block: Depends Verified Bot

View Record Open in Edge BotDB

TikTokSpider Safe

TikTokSpider

TikTokSpider is a web scraper from ByteDance used for public web data collection, page extraction, content monitoring.

Operator and family: ByteDance Type: Scraper Purpose: Scraping

Verified robots.txt: Yes Block: Depends Verified Bot

View Record Open in Edge BotDB

Timpibot Neutral

Timpibot

Timpibot is an AI crawler from Timpi used for AI crawler discovery, assistant retrieval, AI search support, or model-related web access.

Operator and family: Timpi Type: Ai Purpose: AI Data Collection

Not verified robots.txt: No Block: Depends Observed

View Record Open in Edge BotDB

Bytespider Safe

Toutiao

Toutiao is a search crawler from ByteDance used for search indexing, URL discovery, page rendering, result freshness.

Operator and family: ByteDance Type: Search Purpose: Indexing

Verified robots.txt: Yes Block: Depends Verified Bot

View Record Open in Edge BotDB

Trae Neutral

Trae

Trae is an AI crawler used for assistant-driven browsing, page retrieval.

Operator and family: Trae Type: Ai Purpose: Ai Assistant

Not verified robots.txt: Unknown Block: Depends Observed

View Record Open in Edge BotDB

Trellis-Services Neutral

Trellis-Services

Trellis-Services is a web scraper from Mediavine used for public web data collection, page extraction, content monitoring.

Operator and family: Mediavine Type: Scraper Purpose: Scraping

Verified robots.txt: Unknown Block: Depends Verified Bot

View Record Open in Edge BotDB

trendictionbot Safe

Trendiction Bot

Trendiction Bot is a monitoring bot from Trendiction S.A. used for uptime monitoring, synthetic checks, performance measurement, accessibility review.

Operator and family: Trendiction S.A. Type: Monitoring

Verified robots.txt: No Block: No Verified Bot

View Record Open in Edge BotDB

TrustedSite Neutral

TrustedSite

TrustedSite is a security scanner used for security scanning, malware checks, vulnerability assessment, certificate review.

Operator and family: TrustedSite Type: Security Purpose: Site Validation

Not verified robots.txt: Unknown Block: Depends Documented

View Record Open in Edge BotDB

TrustLink-VerificationAgent Neutral

TrustLink VerificationAgent is a verification from Pearl Industry Network LLC used for TrustLink VerificationAgent is used for TrustLink VerificationAgent checks public pages or social profiles for owner-placed verification codes.

Operator: Pearl Industry Network LLC Family: TrustLink Type: Verification Purpose: Ownership Verification

Verified robots.txt: Unknown Block: Depends Documented

View Record Open in Edge BotDB

Trustly Safe

Trustly

Trustly is a webhook callback from Trustly Group AB used for webhook delivery, payment notifications, service callbacks.

Operator and family: Trustly Group AB Type: Webhook

Verified robots.txt: Unknown Block: No Verified Bot

View Record Open in Edge BotDB

TTD-Content Neutral

TTD Content

TTD Content is a search crawler from The Trade Desk used for search indexing, URL discovery, page rendering, result freshness.

Operator and family: The Trade Desk Type: Search Purpose: Indexing

Verified robots.txt: No Block: Depends Verified Bot

View Record Open in Edge BotDB

Tumblr Neutral

Tumblr

Tumblr is a web scraper from Automattic used for public web data collection, page extraction, content monitoring.

Operator and family: Automattic Type: Scraper Purpose: Scraping

Verified robots.txt: Unknown Block: Depends Verified Bot

View Record Open in Edge BotDB

Turnitin Neutral

TurnitinBot

TurnitinBot is a web scraper from Turnitin used for public web data collection, page extraction, content monitoring.

Operator and family: Turnitin Type: Scraper Purpose: Scraping

Verified robots.txt: Yes Block: Depends Verified Bot

View Record Open in Edge BotDB

TurnitinBot/ContentIngest Safe

TurnitinBot ContentIngest

TurnitinBot ContentIngest is a search crawler from Turnitin used for search indexing, URL discovery, page rendering, result freshness.

Operator and family: Turnitin Type: Search Purpose: Indexing

Verified robots.txt: Yes Block: Depends Official Documented

View Record Open in Edge BotDB

TwilioProxy Safe

Twilio Proxy

Twilio Proxy is a webhook callback from Twilio, Inc. used for webhook delivery, payment notifications, service callbacks.

Operator: Twilio, Inc. Family: Inc. Type: Webhook

Verified robots.txt: Unknown Block: No Verified Bot

View Record Open in Edge BotDB

TwinAgent Neutral

TwinAgent

TwinAgent is an AI crawler from Twin used for assistant-driven browsing, page retrieval.

Operator and family: Twin Type: Ai Purpose: Ai Assistant

Verified robots.txt: Unknown Block: Depends Verified Bot

View Record Open in Edge BotDB

Twingly Caution

Twingly

Twingly is a web scraper used for public web data collection, page extraction, content monitoring.

Operator and family: Twingly Type: Content Crawler Purpose: Content Discovery

Verified robots.txt: Unknown Block: Monitor Documented

View Record Open in Edge BotDB

Twitterbot Neutral

twitterbot

twitterbot is a web scraper from Twitter used for public web data collection, page extraction, content monitoring.

Operator and family: Twitter Type: Scraper Purpose: Scraping

Verified robots.txt: Unknown Block: Depends Verified Bot

View Record Open in Edge BotDB

upday Neutral

upday

upday is a search crawler from upday GmbH & Co. KG used for search indexing, URL discovery, page rendering, result freshness.

Operator: upday GmbH & Co. KG Family: upday GmbH & Co. KG Type: Search Purpose: Indexing

Verified robots.txt: Yes Block: Depends Verified Bot

View Record Open in Edge BotDB