Bot intelligence record
Crawlspace
Review firstUse the Crawlspace identifier to separate Crawlspace AI-related crawler, assistant, or retrieval traffic from normal visitor requests in server logs.
- Operator
- Crawlspace
- Family
- Crawlspace
- Type
- Ai
- Source type
- Observed
- Last checked
- 2026-06-20
User-Agent Pattern
CrawlspaceCrawlspace
User-agent strings are identification signals, not proof of identity. Confirm important allow, block, or rate-limit decisions with logs, DNS or IP evidence, request behavior, or operator documentation when available.
Robots.txt Snippet
Click snippet to copyUser-agent: Crawlspace Disallow: /Click the snippet to copy it, or highlight the text manually.
Handling Guidance
DependsUse this record as bot intelligence, then verify the request source and behavior before allowing, blocking, or rate limiting.
AI crawling, assistant retrieval, AI search, security scanning, or model-supporting discovery depending on the user-agent.
Record Details
Structured data- Operator
- Crawlspace
- Family
- Crawlspace
- Type
- Ai
- Purpose
- Scraping
- Identity type
- Observed
- Confidence
- Medium
- Last verified
- 2026-06-20
- Last checked
- 2026-06-20
- Source type
- Observed
- Verification
- Observed public robots token; verify with server logs and operator documentation before allow-listing.
- Spoofing risk
- User-agent strings for Crawlspace can be spoofed. Treat this observed identifier as a classification signal only; verify with logs, request behavior, network origin, and operator documentation before allow-listing or creating low-friction rules.
Notes
Crawlspace is listed in the Botcrawl directory as a crawler associated with Crawlspace. The primary identifier for log review is Crawlspace.
Identification
- User-agent pattern:
Crawlspace - Operator: Crawlspace
- Type: ai
- Kind: crawler
- Purpose: scraping
- Confidence: medium
Common use
Crawlspace crawler identifier associated with web data extraction or scraping workflows.
Verification and handling
Match the identifier as a case-insensitive substring in HTTP user-agent logs. Do not treat a user-agent match alone as proof of identity for allow-listing.
Robots.txt handling: unknown.
Evidence and Source
- Observed public robots token; verify with server logs and operator documentation before allow-listing.
- Match `Crawlspace` as a case-insensitive substring in HTTP user-agent logs. Review bot_aliases and bot_http_agent when present. Do not treat a user-agent match alone as proof of identity for allow-listing.
- AI crawling, assistant retrieval, AI search, security scanning, or model-supporting discovery depending on the user-agent.
- User-agent strings for Crawlspace can be spoofed. Treat this observed identifier as a classification signal only; verify with logs, request behavior, network origin, and operator documentation before allow-listing or creating low-friction rules.
https://github.com/ai-robots-txt/ai.robots.txt/blob/main/robots.txt
Monitor This Bot In Edge
Botcrawl EdgeUse Botcrawl Edge to see matching traffic, create allow or block rules, and control this bot across connected sites.