SEO
37AuditsBot
37AuditsBot checks websites for technical SEO, security, performance, and AI-readiness issues.
SEO
Agency Analytics Crawler
AgencyAnalytics Site Auditor is a technical SEO crawler identified by `RSiteAuditor`.
SEO
Ahrefs Site Audit
AhrefsSiteAudit crawls configured sites for technical and on-page SEO auditing.
SEO
ApexAuditBot
ApexAuditBot retrieves public pages for user-requested SEO and AI-readiness audits.
SEO
AskyBot
AskyBot crawls customer-authorized sites for content, technical health and AI-search visibility analysis.
SEO
Barkrowler
Barkrowler is Babbar's crawler for link-graph, topical, popularity, and SEO intelligence; it follows robots.txt and crawl-delay directives.
SEO
BLEXBot
BLEXBot is an SEO/backlink crawler used to build fresh link and web-index data for marketing intelligence.
SEO
Botify
Botify is a technical SEO crawler used for site audits and search-performance analysis; robots handling can depend on project configuration.
SEO
BrightEdge Bot
BrightEdge Crawler performs technical SEO site audits and content analysis for BrightEdge Site Audit, ContentIQ, and related products.
SEO
Caliperbot
Caliperbot is a crawler associated with Conductor and used for seo content analytics; candidate traffic can include `Caliperbot/1.0`
SEO
CiteSurgeAuditBot
CiteSurgeAuditBot is a audit crawler associated with CiteSurge and used for ai readiness auditing; candidate traffic can include `CiteSurgeAuditBot`
SEO
Claritybot
Claritybot crawls pages for technical SEO audits and content analysis requested by seoClarity clients.
SEO
ClearscopeBot
ClearscopeBot is a crawler associated with Mushi Labs and used for seo content analysis; candidate traffic can include `Clearscopebot`
SEO
DataForSEO
DataForSEO OnPage uses `RSiteAuditor` by default for technical SEO site audits; both the User-Agent and robots handling can be customized.
SEO
DataForSEO Bot
DataForSeoBot is DataForSEO's backlink-data crawler. It follows robots.txt and is distinct from the `RSiteAuditor` OnPage crawler.
SEO
Deepcrawl
Deepcrawl is now Lumar. Current crawling uses the `lumar` identity while the historical `deepcrawl` token can still appear in supported configurations.
SEO
FlowpaneBot
FlowpaneBot is a verified crawler associated with Flowpane used for site audit. It is commonly identified by `FlowpaneBot`.
SEO
foureff
foureff is a verified crawler associated with 4F used for site audit. It is commonly identified by `4f.at/crawler`.
SEO
GoodForBotsBot
GoodForBotsBot checks website pages and public discovery files for AI-agent access and readability.
SEO
Google Schema Markup Testing Tool
Google Schema Markup Testing Tool is associated with Google and is used for validating schema and structured-data markup for Google Search testing workflows.…