Bot Directory

Search public Botcrawl bot records to identify AI crawlers, search bots, scrapers, scanners, monitoring bots, preview bots, and user-agent patterns.

AI Crawlers Search Bots Scrapers Scanners Monitoring Preview Bots
See these bots hitting your site in real time

The public directory explains known bots and user-agents. Botcrawl Edge uses BotDB plus datacenter intelligence for live classification, rules, and enforcement.

Try Edge Free Open Edge BotDB
998 bots
SearchmetricsBot Neutral

SearchmetricsBot

SearchmetricsBot is an SEO crawler from Searchmetrics / Conductor used for SEO analysis, crawl diagnostics, backlink discovery, technical audit work.

Operator: Searchmetrics / Conductor Family: Searchmetrics Type: Seo Crawler Purpose: Seo Auditing
Not verified robots.txt: Unknown Block: Depends Documented
View Record Open in Edge BotDB
SEBot-WA Safe

SEBot-WA

SEBot-WA is a search crawler from SE Ranking used for search indexing, URL discovery, page rendering, result freshness.

Operator and family: SE Ranking Type: Search Purpose: Indexing
Verified robots.txt: Unknown Block: No Verified Bot
View Record Open in Edge BotDB
SecurityHeaders Safe

SecurityHeaders

SecurityHeaders is a security scanner from Security Headers used for security scanning, malware checks, vulnerability assessment, certificate review.

Operator and family: Security Headers Type: Security
Verified robots.txt: Unknown Block: No Verified Bot
View Record Open in Edge BotDB
SeekportBot Safe

Seekport

Seekport is a search crawler from SISTRIX used for search indexing, URL discovery, page rendering, result freshness.

Operator and family: SISTRIX Type: Search Purpose: Indexing
Verified robots.txt: No Block: Depends Verified Bot
View Record Open in Edge BotDB
Seekr Neutral

Seekr

Seekr is a search crawler used for search indexing, URL discovery, page rendering, result freshness.

Operator and family: Seekr Type: Search Purpose: Indexing
Verified robots.txt: Yes Block: Depends Verified Bot
View Record Open in Edge BotDB
SeekrBot Neutral

SeekrBot

SeekrBot is a search crawler from Seekr used for search indexing, URL discovery, page rendering, result freshness.

Operator and family: Seekr Type: Search Purpose: Indexing
Verified robots.txt: Yes Block: Depends Verified Bot
View Record Open in Edge BotDB
SelectikaScraper Neutral

SelectikaScraper

SelectikaScraper is an AI crawler from Selectika used for AI crawler discovery, assistant retrieval, AI search support, or model-related web access.

Operator and family: Selectika Type: Ai Crawler Purpose: Ai Enrichment
Not verified robots.txt: Unknown Block: Depends Documented
View Record Open in Edge BotDB
SemanticScholarBot Safe

Semantic Scholar Bot

Semantic Scholar Bot is a search crawler from Allen Institute for AI used for search indexing, URL discovery, page rendering, result freshness.

Operator: Allen Institute for AI Family: Semantic Scholar Type: Search Purpose: Academic Search Crawling
Verified robots.txt: Yes Block: No Official Documented
View Record Open in Edge BotDB
Semonto Neutral

Semonto

Semonto is a monitoring bot used for uptime monitoring, synthetic checks, performance measurement, accessibility review.

Operator and family: Semonto Type: Monitoring Purpose: Website Monitoring
Not verified robots.txt: Yes Block: Depends Documented
View Record Open in Edge BotDB
SemrushBot-SA Neutral

Semrushbot

Semrushbot is a search crawler from Semrush used for search indexing, URL discovery, page rendering, result freshness.

Operator and family: Semrush Type: Search Purpose: Indexing
Verified robots.txt: Yes Block: Depends Verified Bot
View Record Open in Edge BotDB
SemrushBot Neutral

SemrushBot Core

SemrushBot Core is a search crawler from Semrush used for search indexing, URL discovery, page rendering, result freshness.

Operator and family: Semrush Type: Search Purpose: Indexing
Verified robots.txt: Yes Block: Depends Verified Bot
View Record Open in Edge BotDB
SemrushBot-OCOB Neutral

SemrushBot-OCOB

SemrushBot-OCOB is an AI training crawler from Semrush used for AI model training, dataset discovery.

Operator and family: Semrush Type: Ai Purpose: Ai Training
Verified robots.txt: Yes Block: Depends Verified Bot
View Record Open in Edge BotDB
SemrushBot-BA Neutral

SemrushBotBacklinkAudit

SemrushBotBacklinkAudit is a search crawler from SEMrush used for search indexing, URL discovery, page rendering, result freshness.

Operator and family: SEMrush Type: Search Purpose: Indexing
Verified robots.txt: Yes Block: Depends Verified Bot
View Record Open in Edge BotDB
SemrushBot-SI Neutral

SemrushBotSI

SemrushBotSI is a search crawler from Semrush used for search indexing, URL discovery, page rendering, result freshness.

Operator and family: Semrush Type: Search Purpose: Indexing
Verified robots.txt: Yes Block: Depends Verified Bot
View Record Open in Edge BotDB
SemrushBot-SWA Neutral

SemrushBotSwa

SemrushBotSwa is an AI training crawler from SEMrush used for AI model training, dataset discovery.

Operator and family: SEMrush Type: Ai Purpose: Ai Training
Verified robots.txt: Yes Block: Depends Verified Bot
View Record Open in Edge BotDB
Sendcloud Neutral

Sendcloud

Sendcloud is a service Client used for Sendcloud is used for Sendcloud may make automated HTTP requests for shipping, logistics, integration.

Operator and family: Sendcloud Type: Service Client Purpose: Shipping Automation
Not verified robots.txt: Unknown Block: Depends Documented
View Record Open in Edge BotDB
SendGrid Event API Safe

SendGrid

SendGrid is a webhook callback used for webhook delivery, payment notifications, service callbacks.

Operator and family: SendGrid Type: Webhook
Verified robots.txt: Unknown Block: No Verified Bot
View Record Open in Edge BotDB
SentiBot Neutral

SentiBot

SentiBot is a monitoring bot from SentiOne used for uptime monitoring, synthetic checks, performance measurement, accessibility review.

Operator and family: SentiOne Type: Monitoring Purpose: Social Listening
Not verified robots.txt: Yes Block: Depends Documented
View Record Open in Edge BotDB
SentinelBot Neutral

SentinelBot

SentinelBot is a monitoring bot from rootstuff used for uptime monitoring, synthetic checks, performance measurement, accessibility review.

Operator: rootstuff Family: SentinelBot Type: Monitoring Purpose: Uptime Monitoring
Not verified robots.txt: Unknown Block: Depends Documented
View Record Open in Edge BotDB
https://sentry.io Safe

Sentry

Sentry is a monitoring bot used for uptime monitoring, synthetic checks, performance measurement, accessibility review.

Operator and family: Sentry Type: Monitoring
Verified robots.txt: No Block: No Verified Bot
View Record Open in Edge BotDB
SentryUptimeBot Safe

Sentry Uptime Monitoring

Sentry Uptime Monitoring is a monitoring bot from Sentry Uptime Bot used for uptime monitoring, synthetic checks, performance measurement, accessibility review.

Operator and family: Sentry Uptime Bot Type: Monitoring
Verified robots.txt: No Block: No Verified Bot
View Record Open in Edge BotDB
seo-audit-check-bot Neutral

SEO Audit Check Bot

SEO Audit Check Bot is a web scraper from SEO Audit Check used for public web data collection, page extraction, content monitoring.

Operator and family: SEO Audit Check Type: Scraper Purpose: Scraping
Verified robots.txt: Unknown Block: Depends Verified Bot
View Record Open in Edge BotDB
seo4ajax.com Neutral

seo4ajax

seo4ajax is a search crawler from Capsule Code used for search indexing, URL discovery, page rendering, result freshness.

Operator and family: Capsule Code Type: Search Purpose: Indexing
Verified robots.txt: No Block: Depends Verified Bot
View Record Open in Edge BotDB
SeobilityBot Neutral

SeobilityBot

SeobilityBot is a search crawler from Seobility used for search indexing, URL discovery, page rendering, result freshness.

Operator and family: Seobility Type: Search Purpose: Indexing
Verified robots.txt: No Block: Depends Verified Bot
View Record Open in Edge BotDB