Apify Website Content Crawler

ApifyWebsiteContentCrawler

User-agent strings are identity signals, not proof of identity. Confirm the source and behavior before trusting a request.

Verification method: Verify Apify Website Content Crawler by matching `ApifyWebsiteContentCrawler` to Apify evidence, then checking reverse DNS, source-network ownership, signed request data, or published crawler documentation when available.

Apify Website Content Crawler traffic is primarily detected by the `ApifyWebsiteContentCrawler` user-agent pattern; related patterns include `Apify Website Content Crawler`; a representative HTTP user-agent is `ApifyWebsiteContentCrawler/1.0 (+https://apify.com/apify/website-content-crawler)`. Compare source IPs, reverse DNS, request paths, and crawl cadence with Apify infrastructure before trusting the traffic.

Use this record as identity context. Verify the request source, behavior, and relevance to your site before allowing or restricting traffic.

Apify Website Content Crawler is used for RSS or Atom feed polling, syndication refresh, subscription delivery, and content update checks.

Aliases

Apify Website Content Crawler

Type

feed

Operator

Apify

Family

Apify

Kind

Fetcher

Purpose

Feed Fetch

Identity

Verified Bot

Detection confidence

Medium

Spoofing risk

Apify Website Content Crawler Has Medium Spoofing Risk Because User Agent Strings Can Be Copied; Pair The Match With DNS, IP, Behavior, Or Operator Evidence.

Status

Active

Respects robots.txt

No

Last verified

2026 06 23

User-agent: ApifyWebsiteContentCrawler
Disallow: /

• Source reference
• Operator reference