Overview
Rapidflare Ingestion Crawler is a knowledge ingestion fetcher associated with Rapidflare and used for customer knowledge ingestion.
No fixed HTTP User-Agent is recorded for this identity. Use the cryptographic signed-agent identity when present.
Directory status: Active. Identity classification: Verified with High confidence. Identity type: http message signature.
robots.txt behavior is currently unconfirmed.
Risk classification: Neutral. Evaluate the request context and site policy before allowing, rate-limiting, or blocking.
Identity
- Identity Type
- HTTP message signature
- Evidence Method
- Validate Rapidflare’s RFC 9421 HTTP Message Signature / Web Bot Auth identity (`https://docs.rapidflare.ai`) and optionally corroborate source traffic against the published crawler egress addresses. No fixed User-Agent is required.
Classification
- Type
- AI
- Kind
- Knowledge ingestion fetcher
- Family
- Rapidflare
- Purpose
- Customer knowledge ingestion
Behavior and handling
- Common Use
- fetching websites, sitemaps, and URL lists that customers connect as knowledge sources
- Detection Notes
- No fixed User-Agent is stable for this ingestion identity. Verify the signed-agent identity and HTTP message signature; Rapidflare also publishes a small crawler egress set for corroboration.
- Respects robots.txt
- Unknown
- Spoofing Risk
- Cryptographic request validation provides the primary identity signal. Unsigned lookalike traffic should not be treated as this identity.
- Risk
- Neutral
- Recommended Handling
- Depends
Rules and controls
- Robots.txt Snippet
-
# robots.txt behavior is unconfirmed. Use server-side controls if restriction is required.
Relationships
- Operator
- Rapidflare Checked 2026-10-06
Relationships without an Evidence link are normalized from the canonical directory record. They should not be interpreted as independent proof of physical presence or request origin.