About this bot
DataForSEO is associated with DataForSEO and used for technical seo site auditing.
The primary identifying User-Agent token is RSiteAuditor. Representative HTTP request identity: Mozilla/5.0 (compatible; RSiteAuditor).
Directory status: Active. Identity classification: Verified with High confidence.
robots.txt behavior: depends under RSiteAuditor.
Crawls customer-selected websites for DataForSEO OnPage technical SEO auditing and structured page analysis.
How to identify this bot
- User-agent and crawler identifiers
-
RSiteAuditorMozilla/5.0 (compatible; RSiteAuditor) - Also known as
- DataForSEO OnPage Crawler; DataForSEO Site Auditor DataForSEO OnPage Crawler
Details
- Kind
- Site audit crawler
- Family
- DataForSEO
- Purpose
- Technical SEO site auditing
- Common Use
- Crawls customer-selected websites for DataForSEO OnPage technical SEO auditing and structured page analysis.
- Respects robots.txt
- Depends
Verification
- Identity Type
- Officially documented
- Evidence Method
- Match `RSiteAuditor` as the default DataForSEO OnPage identity and corroborate with DataForSEO's published crawling network where necessary. Customers can replace the User-Agent.
- Detection Notes
- `RSiteAuditor` is the default OnPage UA, but DataForSEO customers can configure another User-Agent and can alter robots handling.
Website controls
- Robots.txt Snippet
# robots.txt handling depends on the request context or service configuration. Use server-side controls when deterministic restriction is required.- Apache Rule
RewriteCond %{HTTP_USER_AGENT} "RSiteAuditor" [NC] RewriteRule ^ - [F,L]- Nginx Rule
if ($http_user_agent ~* "RSiteAuditor") { return 403; }