Overview
Known Agent is an AI crawler from Known Agents used for assistant-driven browsing, page retrieval, and automated web access on behalf of an AI product.
Its primary user-agent pattern is Known Agent; related patterns include Known Agent/1.0; a representative HTTP user-agent is Mozilla/5.0 (compatible; Known Agent/1.0; +https://knownagents.com).
Known Agent is verified with Medium confidence. The identity type is Verified Bot, and the evidence basis is a public crawler reference or source-linked documentation.
Known Agent is marked as not reliably governed by robots.txt directives; use server-side rules if the traffic should be restricted.
Known Agent should be handled according to the site owner’s AI crawler policy, with allow, block, or rate-limit rules applied deliberately.
Identity
- User-Agent
Known Agent- Aliases
- Known Agent/1.0
- HTTP Agent Examples
Mozilla/5.0 (compatible; Known Agent/1.0; +https://knownagents.com)- Robots.txt Token
Known Agent- Identity Type
- Verified Bot
- Evidence Method
- Verify Known Agent by matching `Known Agent` to Known Agents evidence, then checking reverse DNS, source-network ownership, signed request data, or published crawler documentation when available.
Classification
- Type
- AI
- Kind
- Agent
- Family
- Known Agents
- Purpose
- ai-assistant
Behavior and handling
- Common Use
- Known Agent is used for assistant-driven browsing, page retrieval, and automated web access on behalf of an AI product.
- Detection Notes
- Known Agent traffic is primarily detected by the `Known Agent` user-agent pattern; related patterns include `Known Agent/1.0`; a representative HTTP user-agent is `Mozilla/5.0 (compatible; Known Agent/1.0; +https://knownagents.com)`. Compare source IPs, reverse DNS, request paths, and crawl cadence with Known Agents infrastructure before trusting the traffic.
- Respects robots.txt
- No
- Spoofing Risk
- Known Agent has medium spoofing risk because user-agent strings can be copied; pair the match with DNS, IP, behavior, or operator evidence.
- Risk
- Neutral
- Recommended Handling
- Depends
Rules and controls
- Robots.txt Snippet
# This agent may ignore robots.txt. Use authenticated access controls or network policy when blocking is required.