Bot intelligence record
Scrapy
Review firstScrapy is a web scraper used for public web data collection, page extraction, content monitoring; it appears in server logs as `Scrapy`.
User-Agent Pattern
ScrapyScrapy
User-agent strings are identification signals, not proof of identity. Confirm important allow, block, or rate-limit decisions with logs, DNS or IP evidence, request behavior, or operator documentation when available.
Robots.txt Snippet
Click snippet to copyUser-agent: Scrapy
Disallow: /Click the snippet to copy it, or highlight the text manually.
Handling Guidance
DependsUse this record as bot intelligence, then verify the request source and behavior before allowing, blocking, or rate limiting.
Scrapy is used for public web data collection, page extraction, content monitoring, and third-party crawler activity.
Record Details
Verified fields- Operator
- Scrapy
- Family
- Scrapy
- Type
- Scraper
- Purpose
- Web Scraping
- Identity type
- Official Documented
- Confidence
- High
- Last verified
- 2026-06-22
- Last checked
- 2026-06-22
- Source type
- Official
- Verification
- Verify Scrapy by matching `Scrapy` to official operator documentation, then checking reverse DNS, IP ownership, request behavior, and crawl consistency.
- Spoofing risk
- Scrapy has medium spoofing risk because the user-agent can be copied, even when the bot has strong source or documentation support.
Notes
- Scrapy is a web scraper used for public web data collection, page extraction, content monitoring, and third-party crawler activity.
- Its primary user-agent pattern is
Scrapy; related patterns includeScrapy/VERSION (+https://scrapy.org); a representative HTTP user-agent isScrapy/VERSION (+https://scrapy.org). - Scrapy is verified with High confidence. The identity type is Official Documented, and the evidence basis is official operator documentation.
- Scrapy does not have confirmed robots.txt behavior in the available public evidence.
- Scrapy should be reviewed against site policy, source evidence, crawl rate, and requested paths before a permanent allow or block rule is created.
Evidence and Source
- Verify Scrapy by matching `Scrapy` to official operator documentation, then checking reverse DNS, IP ownership, request behavior, and crawl consistency.
- Scrapy traffic is primarily detected by the `Scrapy` user-agent pattern; related patterns include `Scrapy/VERSION (+https://scrapy.org)`; a representative HTTP user-agent is `Scrapy/VERSION (+https://scrapy.org)`. Compare source IPs, reverse DNS, request paths, and crawl cadence before trusting the traffic.
- Scrapy is used for public web data collection, page extraction, content monitoring, and third-party crawler activity.
- Scrapy has medium spoofing risk because the user-agent can be copied, even when the bot has strong source or documentation support.
https://docs.scrapy.org/en/latest/topics/settings.html#user-agent
Use This Signal In Edge
Botcrawl EdgeUse Edge to match live requests, detect related datacenter traffic, and apply allow, block, rate-limit, or log rules.