Bot intelligence record
ApifyBot
Review firstUse the ApifyBot identifier to separate Apify AI-related crawler, assistant, or retrieval traffic from normal visitor requests in server logs.
- Operator
- Apify
- Family
- Apify
- Type
- Ai
- Source type
- Observed
- Last checked
- 2026-06-20
User-Agent Pattern
ApifyApifyBot
User-agent strings are identification signals, not proof of identity. Confirm important allow, block, or rate-limit decisions with logs, DNS or IP evidence, request behavior, or operator documentation when available.
Robots.txt Snippet
Click snippet to copyUser-agent: ApifyBot Disallow: /Click the snippet to copy it, or highlight the text manually.
Handling Guidance
DependsUse this record as bot intelligence, then verify the request source and behavior before allowing, blocking, or rate limiting.
AI crawling, assistant retrieval, AI search, security scanning, or model-supporting discovery depending on the user-agent.
Record Details
Structured data- Operator
- Apify
- Family
- Apify
- Type
- Ai
- Purpose
- Scraping
- Identity type
- Observed
- Confidence
- Low
- Last verified
- 2026-06-20
- Last checked
- 2026-06-20
- Source type
- Observed
- Verification
- Observed public robots token; verify with server logs and operator documentation before allow-listing.
- Spoofing risk
- User-agent strings for ApifyBot can be spoofed. Treat this observed identifier as a classification signal only; verify with logs, request behavior, network origin, and operator documentation before allow-listing or creating low-friction rules.
Notes
ApifyBot is listed in the Botcrawl directory as a crawler associated with Apify. The primary identifier for log review is ApifyBot.
Identification
- User-agent pattern:
ApifyBot - Operator: Apify
- Type: ai
- Kind: crawler
- Purpose: scraping
- Confidence: low
Common use
Apify-related crawler identifier used for automated extraction or content retrieval workflows.
Verification and handling
Match the identifier as a case-insensitive substring in HTTP user-agent logs. Do not treat a user-agent match alone as proof of identity for allow-listing.
Robots.txt handling: unknown.
Evidence and Source
- Observed public robots token; verify with server logs and operator documentation before allow-listing.
- Match `ApifyBot` as a case-insensitive substring in HTTP user-agent logs. Review bot_aliases and bot_http_agent when present. Do not treat a user-agent match alone as proof of identity for allow-listing.
- AI crawling, assistant retrieval, AI search, security scanning, or model-supporting discovery depending on the user-agent.
- User-agent strings for ApifyBot can be spoofed. Treat this observed identifier as a classification signal only; verify with logs, request behavior, network origin, and operator documentation before allow-listing or creating low-friction rules.
https://github.com/ai-robots-txt/ai.robots.txt/blob/main/robots.txt
Monitor This Bot In Edge
Botcrawl EdgeUse Botcrawl Edge to see matching traffic, create allow or block rules, and control this bot across connected sites.