Bot intelligence record
FirecrawlAgent
Review firstUse the FirecrawlAgent identifier to separate Firecrawl AI-related crawler, assistant, or retrieval traffic from normal visitor requests in server logs.
- Operator
- Firecrawl
- Family
- Firecrawl
- Type
- Ai
- Source type
- Official
- Last checked
- 2026-06-20
User-Agent Pattern
FirecrawlFirecrawlAgent
User-agent strings are identification signals, not proof of identity. Confirm important allow, block, or rate-limit decisions with logs, DNS or IP evidence, request behavior, or operator documentation when available.
Robots.txt Snippet
Click snippet to copyUser-agent: FirecrawlAgent Disallow: /Click the snippet to copy it, or highlight the text manually.
Handling Guidance
DependsUse this record as bot intelligence, then verify the request source and behavior before allowing, blocking, or rate limiting.
Web scraping, page extraction, site crawling, and content conversion for AI applications.
Record Details
Structured data- Operator
- Firecrawl
- Family
- Firecrawl
- Type
- Ai
- Purpose
- Scraping
- Identity type
- Official Documented
- Confidence
- High
- Last verified
- 2026-06-20
- Last checked
- 2026-06-20
- Source type
- Official
- Verification
- Official product documentation references the FirecrawlAgent robots.txt directive.
- Spoofing risk
- User-agent strings for FirecrawlAgent can be spoofed. Treat the published identifier as a classification signal, then verify with official documentation, IP ranges, reverse DNS, signed request headers, or other operator-backed trust signals before allow-listing.
Notes
FirecrawlAgent is listed in the Botcrawl directory as a crawler associated with Firecrawl. The primary identifier for log review is FirecrawlAgent.
Identification
- User-agent pattern:
FirecrawlAgent - Operator: Firecrawl
- Type: ai
- Kind: crawler
- Purpose: scraping
- Confidence: high
Common use
FirecrawlAgent is used by Firecrawl for search, scrape, crawl, and interaction workflows that convert web pages into structured data.
Verification and handling
Match the identifier as a case-insensitive substring in HTTP user-agent logs. Do not treat a user-agent match alone as proof of identity for allow-listing.
Robots.txt handling: yes.
Evidence and Source
- Official product documentation references the FirecrawlAgent robots.txt directive.
- Match `FirecrawlAgent` as a case-insensitive substring in HTTP user-agent logs. Review bot_aliases and bot_http_agent when present. Do not treat a user-agent match alone as proof of identity for allow-listing.
- Web scraping, page extraction, site crawling, and content conversion for AI applications.
- User-agent strings for FirecrawlAgent can be spoofed. Treat the published identifier as a classification signal, then verify with official documentation, IP ranges, reverse DNS, signed request headers, or other operator-backed trust signals before allow-listing.
Monitor This Bot In Edge
Botcrawl EdgeUse Botcrawl Edge to see matching traffic, create allow or block rules, and control this bot across connected sites.