Overview
InternetArchiveBot is a feed fetcher from Internet Archive used for RSS or Atom feed polling, syndication refresh, subscription delivery, and content update checks.
Its primary user-agent pattern is IABot; related patterns include IABot/; a representative HTTP user-agent is IABot/2.0 (+https://meta.wikimedia.org/wiki/InternetArchiveBot/FAQ_for_sysadmins) (Checking if link from Wikipedia is broken and needs removal).
InternetArchiveBot is Unverified at the identity-evidence level. The listed identity remains useful for detection, but this record does not currently contain authoritative evidence sufficient to authenticate the identity claim.
InternetArchiveBot is marked as not reliably governed by robots.txt directives; use server-side rules if the traffic should be restricted.
InternetArchiveBot can usually be allowed after confirming the source and monitoring request volume.
Identity
- User-Agent
IABot- Aliases
- IABot/
- HTTP Agent Examples
IABot/2.0 (+https://meta.wikimedia.org/wiki/InternetArchiveBot/FAQ_for_sysadmins) (Checking if link from Wikipedia is broken and needs removal)- Robots.txt Token
IABot- Identity Type
- Observed
- Evidence Method
- Treat `IABot` as an identity signal only. Confirm it with current operator documentation, cryptographic verification, forward-confirmed reverse DNS, source-network ownership, or other authoritative evidence before trusting the claimed identity.
Classification
- Type
- Feed
- Kind
- Fetcher
- Family
- Internet Archive
- Purpose
- feed-fetch
Behavior and handling
- Common Use
- InternetArchiveBot is used for RSS or Atom feed polling, syndication refresh, subscription delivery, and content update checks.
- Detection Notes
- InternetArchiveBot traffic is primarily detected by the `IABot` user-agent pattern; related patterns include `IABot/`; a representative HTTP user-agent is `IABot/2.0 (+https://meta.wikimedia.org/wiki/InternetArchiveBot/FAQ_for_sysadmins) (Checking if link from Wikipedia is broken and needs removal)`. Compare source IPs, reverse DNS, request paths, and crawl cadence with Internet Archive infrastructure before trusting the traffic.
- Respects robots.txt
- No
- Spoofing Risk
- InternetArchiveBot has medium spoofing risk because user-agent strings can be copied; pair the match with DNS, IP, behavior, or operator evidence.
- Risk
- Safe
- Recommended Handling
- No
Rules and controls
- Robots.txt Snippet
# This agent may ignore robots.txt. Use authenticated access controls or network policy when blocking is required.