Overview
Ai2Bot-Dolma is a training crawler associated with Allen Institute for AI (Ai2) and used for open language model training.
The primary identifying User-Agent token is Ai2Bot-Dolma. Representative HTTP request identity: Mozilla/5.0 (compatible) Ai2Bot-Dolma (+https://www.allenai.org/crawler).
Directory status: Active. Identity classification: Unverified with Medium confidence. Identity type: observed.
robots.txt behavior is currently unconfirmed.
Risk classification: Neutral. Evaluate request context and site policy before allowing, rate-limiting, or blocking.
Identity
- User-Agent Pattern
-
Ai2Bot-Dolma - HTTP Agent Examples
-
Mozilla/5.0 (compatible) Ai2Bot-Dolma (+https://www.allenai.org/crawler) - Robots Token
- Ai2Bot-Dolma
- Identity Type
- Observed
- Evidence Method
- Treat `Ai2Bot-Dolma` as an observed identification signal; current evidence is insufficient to authenticate origin from the User-Agent alone.
Classification
- Type
- AI
- Kind
- Training crawler
- Family
- Ai2
- Purpose
- Open language model training
Behavior and handling
- Common Use
- observed web-corpus collection associated with Ai2 Dolma and open-language-model training workflows
- Detection Notes
- Match the `Ai2Bot-Dolma` User-Agent identity.
- Respects robots.txt
- Unknown
- Spoofing Risk
- The `Ai2Bot-Dolma` User-Agent can be copied by unrelated clients. Treat it as a claimed identity unless corroborated by stronger signals.
- Risk
- Neutral
- Recommended Handling
- Depends
Rules and controls
- Robots.txt Snippet
-
# robots.txt behavior is unconfirmed. Use server-side controls if restriction is required.