About this bot
Crawlspace is associated with Crawlspace and used for user configured web crawling.
The primary identifying User-Agent token is Crawlspace.
Directory status: Active. Identity classification: Verified with High confidence.
It is documented to honor applicable robots.txt directives using Crawlspace.
Risk classification: Neutral. Recommended handling: depends.
How to identify this bot
- User-agent and crawler identifiers
-
Crawlspace
Details
- Kind
- Crawler
- Family
- Crawlspace
- Purpose
- User configured web crawling
- Common Use
- Runs user-authored crawlers for structured web collection, scraping, embedding, and related workflows.
- Respects robots.txt
- Yes
Verification
- Identity Type
- platform-family
- Evidence Method
- Match the immutable `Crawlspace` UA prefix and evaluate the embedded author UUID; individual crawlers are user-authored deployments.
- Detection Notes
- Crawlspace outbound requests use a platform-set User-Agent prefixed with `Crawlspace` and containing the crawler author's UUID.
Website controls
- Robots.txt Snippet
User-agent: Crawlspace Disallow: /