Overview
ShapBot is a crawler associated with Parallel Web Systems, used for building and maintaining Parallel's web search index.
Its primary user-agent pattern is ShapBot; related patterns include ShapBot/0.1.0; a representative HTTP user-agent is Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ShapBot/0.1.0.
ShapBot is Unverified at the identity-evidence level. The listed identity remains useful for detection, but this record does not currently contain authoritative evidence sufficient to authenticate the identity claim.
ShapBot is documented or strongly corroborated as honoring robots.txt directives.
Parallel publishes the full user-agent, robots.txt token, purpose, and a current IP-range JSON file.
Treat the user-agent as an identity claim. Corroborate it with operator-controlled IP ranges, reverse DNS, signed-request evidence, or current documentation when available.
Identity
- User-Agent
ShapBot- Aliases
- ShapBot/0.1.0
- HTTP Agent Examples
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ShapBot/0.1.0- Robots.txt Token
ShapBot- Identity Type
- Observed
- Evidence Method
- Treat `ShapBot` as an identity signal only. Confirm it with current operator documentation, cryptographic verification, forward-confirmed reverse DNS, source-network ownership, or other authoritative evidence before trusting the claimed identity.
Classification
- Type
- Search
- Kind
- Crawler
- Family
- Parallel
- Purpose
- ai-search-indexing
Behavior and handling
- Common Use
- building and maintaining Parallel's web search index
- Detection Notes
- Detect primarily by `ShapBot`; related patterns include `ShapBot/0.1.0`; representative HTTP user-agent: `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ShapBot/0.1.0`. User-agent strings can be copied, so corroborate with stronger evidence before granting trust.
- Respects robots.txt
- Yes
- Spoofing Risk
- ShapBot has medium spoofing risk because its user-agent can be copied; use published operator-controlled verification data when available.
- Risk
- Neutral
- Recommended Handling
- Depends
Rules and controls
- Robots.txt Snippet
User-agent: ShapBot Disallow: /- Apache Rule
RewriteCond %{HTTP_USER_AGENT} "ShapBot" [NC] RewriteRule ^ - [F,L]- Nginx Rule
if ($http_user_agent ~* "ShapBot") { return 403; }