About this bot
Internet Archive – Archive-It is a crawler-service associated with Internet Archive and used for web-archiving.
The primary identifying token is Archive-It. Representative HTTP request identity: Archive-It / archive.org_bot / partner-configured Archive-It crawler identities.
Directory status: Active. Identity classification: Verified with High confidence.
robots.txt behavior depends on the request mode, crawler, or product configuration and should not be generalized across all traffic in this record.
Archive-It is a hosted web-archiving service. Its crawlers normally respect robots.txt by default, but collection owners can configure robots handling and different crawl engines can behave differently.
How to identify this bot
- User-agent and crawler identifiers
-
Archive-ItArchive-It / archive.org_bot / partner-configured Archive-It crawler identities - Also known as
- archive.org_bot; special_archiver archive.org_bot special_archiver
Details
- Kind
- Crawler service
- Family
- Archive-It
- Purpose
- Web archiving
- Common Use
- Archive-It is a hosted web-archiving service. Its crawlers normally respect robots.txt by default, but collection owners can configure robots handling and different crawl engines can behave differently.
- Respects robots.txt
- Depends
Verification
- Identity Type
- Officially documented
- Evidence Method
- Match `Archive-It` as the documented identity signal and corroborate it with request-context, network, DNS, or cryptographic signals where available.
- Detection Notes
- Match the `Archive-It` identity token in the HTTP User-Agent or documented request identity. User-Agent text alone remains spoofable.
Website controls
- Robots.txt Snippet
# robots.txt behavior depends on the request mode or product configuration. Apply the documented control for the specific traffic class.