Diffbot-User

Review

Diffbot-User is a web scraper from Diffbot used for public web data collection, page extraction, content monitoring.

User-Agent

Diffbot-User

User-agent strings are identity signals, not proof of identity. Confirm the source and behavior before trusting a request.

Verification and Detection

Verification method: Verify Diffbot-User by matching `Diffbot-User` to Diffbot evidence, then checking reverse DNS, source-network ownership, signed request data, or published crawler documentation when available.

Diffbot-User traffic is primarily detected by the `Diffbot-User` user-agent pattern; related patterns include `Diffbot user fetcher`. Compare source IPs, reverse DNS, request paths, and crawl cadence with Diffbot infrastructure before trusting the traffic.

Handling Guidance

Use this record as identity context. Verify the request source, behavior, and relevance to your site before allowing or restricting traffic.

Diffbot-User is used for public web data collection, page extraction, content monitoring, and third-party crawler activity.

Record Details

Aliases: Diffbot User Fetcher
Type: scraper
Operator: Diffbot
Family: Diffbot
Kind: Fetcher
Purpose: Scraping
Identity: Official Documented
Detection confidence: High
Spoofing risk: Diffbot User Has Medium Spoofing Risk Because The User Agent Can Be Copied, Even When The Bot Has Strong Source Or Documentation Support.
Status: Active
Respects robots.txt: Yes
Last verified: 2026 04 29

Robots.txt Snippet

User-agent: Diffbot-User
Disallow: /