Diffbot
ReviewDiffbot is a search crawler used for search indexing, URL discovery, page rendering, result freshness.
User-Agent
DiffbotUser-agent strings are identity signals, not proof of identity. Confirm the source and behavior before trusting a request.
Verification and Detection
Verification method: Verify Diffbot by matching `Diffbot` to official operator documentation, then checking reverse DNS, IP ownership, request behavior, and crawl consistency.
Diffbot traffic is primarily detected by the `Diffbot` user-agent pattern; related patterns include `Diffbot crawler`. Compare source IPs, reverse DNS, request paths, and crawl cadence before trusting the traffic.
Handling Guidance
Use this record as identity context. Verify the request source, behavior, and relevance to your site before allowing or restricting traffic.
Diffbot is used for search indexing, URL discovery, page rendering, result freshness, and search-quality checks.
Record Details
- Aliases
- Diffbot Crawler
- Type
- search
- Operator
- Diffbot
- Family
- Diffbot
- Kind
- Crawler
- Purpose
- Indexing
- Identity
- Official Documented
- Detection confidence
- High
- Spoofing risk
- Diffbot Has Medium Spoofing Risk Because The User Agent Can Be Copied, Even When The Bot Has Strong Source Or Documentation Support.
- Status
- Active
- Respects robots.txt
- Yes
- Last verified
- 2026 04 29
Robots.txt Snippet
User-agent: Diffbot
Disallow: /