Overview
Heritrix is an archive Crawler from Internet Archive / Heritrix used for web archiving, preservation crawls, historical capture, and public record indexing.
Its primary user-agent pattern is heritrix.
Heritrix is Unverified at the identity-evidence level. The listed identity remains useful for detection, but this record does not currently contain authoritative evidence sufficient to authenticate the identity claim.
Robots.txt behavior is not currently confirmed.
Heritrix should be monitored first, then rate-limited or blocked if the crawl rate, paths, or behavior are unwanted.
Identity
- User-Agent
heritrix- HTTP Agent Examples
heritrix- Robots.txt Token
heritrix- Identity Type
- Observed
- Evidence Method
- Treat `heritrix` as an identity signal only. Confirm it with current operator documentation, cryptographic verification, forward-confirmed reverse DNS, source-network ownership, or other authoritative evidence before trusting the claimed identity.
Classification
- Type
- Archive Crawler
- Kind
- Archive Crawler
- Family
- Internet Archive / Heritrix
- Purpose
- web-archiving
Behavior and handling
- Common Use
- Heritrix is used for web archiving, preservation crawls, historical capture, and public record indexing.
- Detection Notes
- Heritrix traffic is primarily detected by the `heritrix` user-agent pattern. Compare source IPs, reverse DNS, request paths, and crawl cadence with Internet Archive / Heritrix infrastructure before trusting the traffic.
- Respects robots.txt
- Unknown
- Spoofing Risk
- Heritrix has medium spoofing risk because user-agent strings can be copied; pair the match with DNS, IP, behavior, or operator evidence.
- Risk
- Neutral
- Recommended Handling
- Monitor
Rules and controls
- Robots.txt Snippet
# robots.txt behavior is unconfirmed. Do not rely on this rule without verification.