About this bot
Internet Archive is a crawler associated with Internet Archive and used for web-archiving.
The primary identifying token is archive.org_bot. Representative HTTP request identity: Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot).
Directory status: Active. Identity classification: Verified with High confidence.
This identity is recorded as honoring robots.txt under archive.org_bot.
Internet Archive uses this crawler identity for web preservation and Wayback Machine archiving; it is not a feed-only fetcher.
How to identify this bot
- User-agent and crawler identifiers
-
archive.org_botMozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot) - Also known as
- Wayback Machine crawler
Details
- Kind
- Crawler
- Family
- Internet Archive
- Purpose
- Web archiving
- Common Use
- Internet Archive uses this crawler identity for web preservation and Wayback Machine archiving; it is not a feed-only fetcher.
- Respects robots.txt
- Yes
Verification
- Identity Type
- Officially documented
- Evidence Method
- Match `archive.org_bot` as the documented identity signal and corroborate it with request-context, network, DNS, or cryptographic signals where available.
- Detection Notes
- Match the `archive.org_bot` identity token in the HTTP User-Agent or documented request identity. User-Agent text alone remains spoofable.
Website controls
- Robots.txt Snippet
User-agent: archive.org_bot Disallow: /