Overview
AI2Bot is an AI crawler from Allen Institute for AI used for AI crawler discovery, assistant retrieval, AI search support, or model-related web access.
Its primary user-agent pattern is AI2Bot; related patterns include Mozilla/5.0 (compatible) AI2Bot (+https://www.allenai.org/crawler); Ai2Bot-Dolma; a representative HTTP user-agent is Mozilla/5.0 (compatible) AI2Bot (+https://www.allenai.org/crawler).
AI2Bot is verified with High confidence. The identity type is Official Documented, and the evidence basis is official operator documentation.
AI2Bot is marked as respecting robots.txt directives for crawler access control.
AI2Bot should be handled according to the site owner’s AI crawler policy, with allow, block, or rate-limit rules applied deliberately.
Identity
- User-Agent Pattern
-
AI2Bot - Aliases
- Mozilla/5.0 (compatible) AI2Bot (+https://www.allenai.org/crawler); Ai2Bot-Dolma
- HTTP Agent Examples
-
Mozilla/5.0 (compatible) AI2Bot (+https://www.allenai.org/crawler) - Robots Token
- AI2Bot
- Identity Type
- Officially documented
- Evidence Method
- Verify AI2Bot by matching `AI2Bot` to Allen Institute for AI evidence, then checking reverse DNS, source-network ownership, signed request data, or published crawler documentation when available.
Classification
- Type
- AI
- Kind
- Crawler
- Family
- Allen Institute for AI
- Purpose
- AI research crawling
Behavior and handling
- Common Use
- AI2Bot is used for AI crawler discovery, assistant retrieval, AI search support, or model-related web access.
- Detection Notes
- AI2Bot traffic is primarily detected by the `AI2Bot` user-agent pattern; related patterns include `Mozilla/5.0 (compatible) AI2Bot (+https://www.allenai.org/crawler); Ai2Bot-Dolma`; a representative HTTP user-agent is `Mozilla/5.0 (compatible) AI2Bot (+https://www.allenai.org/crawler)`. Compare source IPs, reverse DNS, request paths, and crawl cadence with Allen Institute for AI infrastructure before trusting the traffic.
- Respects robots.txt
- Yes
- Spoofing Risk
- AI2Bot has medium spoofing risk because the user-agent can be copied, even when the bot has strong source or documentation support.
- Risk
- Neutral
- Recommended Handling
- Depends
Rules and controls
- Robots.txt Snippet
-
User-agent: AI2Bot Disallow: /
Relationships
- Operator
- Allen Institute for AI Checked 2026-06-02
Relationships without an Evidence link are normalized from the canonical directory record. They should not be interpreted as independent proof of physical presence or request origin.