Overview
anthropic-ai is a crawler associated with Anthropic, used for AI model training, dataset discovery, and collection of public web content for model-development pipelines.
Its primary user-agent pattern is anthropic-ai.
anthropic-ai is unverified with Low confidence. The identity type is Observed. This legacy or noncanonical identity is retained for historical log matching.
Robots.txt behavior for anthropic-ai is not confirmed. Do not treat a user-agent-only rule as guaranteed enforcement.
This identity is not included in the operator’s current published crawler list.
Treat the user-agent or identity as a claim and corroborate it with published network data, reverse DNS, signed-request evidence, or current operator documentation when available.
Identity
- User-Agent Pattern
-
anthropic-ai - Robots Token
- anthropic-ai
- Identity Type
- Observed
- Evidence Method
- Verify anthropic-ai by matching `anthropic-ai` to Anthropic evidence, then corroborating the request with source-network ownership, reverse DNS, signed request data, or current operator documentation.
Classification
- Type
- AI
- Kind
- Crawler
- Family
- Anthropic
- Purpose
- AI training
Behavior and handling
- Common Use
- anthropic-ai is used for AI model training, dataset discovery, and collection of public web content for model-development pipelines.
- Detection Notes
- anthropic-ai traffic is primarily detected by `anthropic-ai`. Treat these values as identification claims and corroborate them with source-network ownership, reverse DNS, signed request data, or current operator documentation.
- Respects robots.txt
- Unknown
- Spoofing Risk
- anthropic-ai can be spoofed if identification relies only on a user-agent string; corroborate the request with stronger operator-controlled evidence when available.
- Risk
- Neutral
- Recommended Handling
- Depends
Rules and controls
- Robots.txt Snippet
-
# Legacy or noncanonical identity. Confirm the current canonical token before relying on robots.txt.
Relationships
- Operator
- Anthropic Checked 2026-08-02
Relationships without an Evidence link are normalized from the canonical directory record. They should not be interpreted as independent proof of physical presence or request origin.