Arquivo Web Crawler

Review

Arquivo Web Crawler is a feed fetcher from Arquivo used for RSS or Atom feed polling, syndication refresh, subscription delivery.

User-Agent

Arquivo-web-crawler

User-agent strings are identity signals, not proof of identity. Confirm the source and behavior before trusting a request.

Verification and Detection

Verification method: Verify Arquivo Web Crawler by matching `Arquivo-web-crawler` to Arquivo evidence, then checking reverse DNS, source-network ownership, signed request data, or published crawler documentation when available.

Arquivo Web Crawler traffic is primarily detected by the `Arquivo-web-crawler` user-agent pattern; a representative HTTP user-agent is `Arquivo-web-crawler (compatible; heritrix/3.4.0-20200304 +https://arquivo.pt/faq-crawling)`. Compare source IPs, reverse DNS, request paths, and crawl cadence with Arquivo infrastructure before trusting the traffic.

Handling Guidance

Use this record as identity context. Verify the request source, behavior, and relevance to your site before allowing or restricting traffic.

Arquivo Web Crawler is used for RSS or Atom feed polling, syndication refresh, subscription delivery, and content update checks.

Record Details

Type: feed
Operator: Arquivo
Family: Arquivo
Kind: Fetcher
Purpose: Feed Fetch
Identity: Verified Bot
Detection confidence: Medium
Spoofing risk: Arquivo Web Crawler Has Medium Spoofing Risk Because User Agent Strings Can Be Copied; Pair The Match With DNS, IP, Behavior, Or Operator Evidence.
Status: Active
Respects robots.txt: Yes
Last verified: 2026 06 23

Robots.txt Snippet

User-agent: Arquivo-web-crawler
Disallow: /