Overview
Cloudflare Robots API is a monitoring and operations fetcher operated by Cloudflare. It retrieves robots.txt files for Cloudflare AI Crawl Control so crawler directives can be evaluated.
Its exact HTTP user agent is CloudflareRobotsAPI/1.0, with the robots token CloudflareRobotsAPI.
Cloudflare Robots API is Unverified at the identity-evidence level. The listed identity remains useful for detection, but this record does not currently contain authoritative evidence sufficient to authenticate the identity claim.
Because its function is to retrieve robots.txt directives, blocking it may prevent Cloudflare AI Crawl Control from evaluating a site’s crawler policy.
Allow it by default when Cloudflare AI Crawl Control is in use; otherwise monitor and verify signed requests before making a policy decision.
Identity
- User-Agent
CloudflareRobotsAPI- Aliases
- Cloudflare Robots API
- HTTP Agent Examples
CloudflareRobotsAPI/1.0- Robots.txt Token
CloudflareRobotsAPI- Identity Type
- Http Message Signature
- Evidence Method
- Treat `CloudflareRobotsAPI` as an identity signal only. Confirm it with current operator documentation, cryptographic verification, forward-confirmed reverse DNS, source-network ownership, or other authoritative evidence before trusting the claimed identity.
Classification
- Type
- Monitoring
- Kind
- Fetcher
- Family
- Cloudflare
- Purpose
- monitoring
Behavior and handling
- Common Use
- Retrieves robots.txt files for Cloudflare AI Crawl Control to evaluate crawler directives.
- Detection Notes
- Prefer cryptographic signature verification over user-agent matching. The declared HTTP user agent is `CloudflareRobotsAPI/1.0`.
- Respects robots.txt
- Depends
- Spoofing Risk
- Low when the HTTP Message Signature is verified; high if relying only on the copyable user-agent string.
- Risk
- Safe
- Recommended Handling
- No
Rules and controls
- Robots.txt Snippet
# Blocking this fetcher can prevent Cloudflare AI Crawl Control from evaluating robots.txt. User-agent: CloudflareRobotsAPI Disallow: /