Overview
AgentTrust Collector is a specialized crawler operated by agenttru.st that discovers publicly advertised AI agents. It begins with hostnames found in public Certificate Transparency logs and requests robots.txt along with a fixed set of well-known AI agent and MCP metadata documents.
According to the operator, the crawler does not scan ports, guess arbitrary paths, follow ordinary website links, execute JavaScript, submit forms, attempt authentication, or perform vulnerability probing. A confirmed agent may be rechecked every seven days, while hosts that do not publish supported discovery documents are revisited less frequently.
AgentTrust Collector honors the agenttrust-collector robots.txt token. The operator also publishes current crawler addresses and recommends forward-confirmed reverse DNS under agenttru.st as the primary verification method.
Identity
- User-Agent Pattern
-
AgentTrust-Collector - Aliases
- AgentTrust-Collector, agenttrust-collector
- HTTP Agent Examples
-
AgentTrust-Collector/1.0 (+https://agenttru.st/crawler; purpose=AI Agent Discovery; contact=admin@agenttru.st) - Robots Token
- agenttrust-collector
- Identity Type
- Officially documented
- Evidence Method
- First-party crawler documentation and forward-confirmed reverse DNS under agenttru.st.
Classification
- Type
- Crawler
- Kind
- AI agent discovery crawler
- Family
- AgentTrust
- Operator
- agenttru.st
- Company
- agenttru.st
- Purpose
- AI agent discovery
Behavior and handling
- Common Use
- Discovering publicly advertised AI agents, agent cards, and MCP server metadata.
- Detection Notes
- Verify the requesting IP with forward-confirmed reverse DNS. The reverse hostname must end in agenttru.st, and resolving that hostname must return the original IP address. Published addresses currently include 51.83.6.2, 74.208.40.151, 54.39.22.255, and 51.38.70.50, but the operator states that the fleet may change.
- Respects robots.txt
- Yes
- Spoofing Risk
- The User-Agent can be copied by other clients. Verify legitimate requests using forward-confirmed reverse DNS under agenttru.st.
- Risk
- Neutral
- Recommended Handling
- Allow
Rules and controls
- Robots.txt Snippet
-
User-agent: agenttrust-collector Disallow: / - Apache Rule
-
RewriteEngine On RewriteCond %{HTTP_USER_AGENT} AgentTrust-Collector [NC] RewriteRule .* - [F,L] - Nginx Rule
-
if ($http_user_agent ~* "AgentTrust-Collector") { return 403; }