Overview
Semantic Scholar Bot is a search crawler from Allen Institute for AI used for search indexing, URL discovery, page rendering, result freshness, and search-quality checks.
Its primary user-agent pattern is SemanticScholarBot; related patterns include Mozilla/5.0 (compatible) SemanticScholarBot (+https://www.semanticscholar.org/crawler); a representative HTTP user-agent is Mozilla/5.0 (compatible) SemanticScholarBot (+https://www.semanticscholar.org/crawler).
Semantic Scholar Bot is verified with High confidence. The identity type is Official Documented, and the evidence basis is official operator documentation.
Semantic Scholar Bot is marked as respecting robots.txt directives for crawler access control.
Semantic Scholar Bot can usually be allowed after confirming the source and monitoring request volume.
Semantic Scholar Bot belongs to the Semantic Scholar bot family.
Identity
- User-Agent
SemanticScholarBot- Aliases
- Mozilla/5.0 (compatible) SemanticScholarBot (+https://www.semanticscholar.org/crawler)
- HTTP Agent Examples
Mozilla/5.0 (compatible) SemanticScholarBot (+https://www.semanticscholar.org/crawler)- Robots.txt Token
SemanticScholarBot- Identity Type
- Official Documented
- Evidence Method
- Verify Semantic Scholar Bot by matching `SemanticScholarBot` to Allen Institute for AI evidence, then checking reverse DNS, source-network ownership, signed request data, or published crawler documentation when available.
Classification
- Type
- Search
- Kind
- Crawler
- Family
- Semantic Scholar
- Purpose
- academic search crawling
Behavior and handling
- Common Use
- Semantic Scholar Bot is used for search indexing, URL discovery, page rendering, result freshness, and search-quality checks.
- Detection Notes
- Semantic Scholar Bot traffic is primarily detected by the `SemanticScholarBot` user-agent pattern; related patterns include `Mozilla/5.0 (compatible) SemanticScholarBot (+https://www.semanticscholar.org/crawler)`; a representative HTTP user-agent is `Mozilla/5.0 (compatible) SemanticScholarBot (+https://www.semanticscholar.org/crawler)`. Compare source IPs, reverse DNS, request paths, and crawl cadence with Allen Institute for AI infrastructure before trusting the traffic.
- Respects robots.txt
- Yes
- Spoofing Risk
- Semantic Scholar Bot has medium spoofing risk because the user-agent can be copied, even when the bot has strong source or documentation support.
- Risk
- Safe
- Recommended Handling
- No
Rules and controls
- Robots.txt Snippet
User-agent: SemanticScholarBot Disallow: /