Overview
xAI-SearchBot is a fetcher associated with xAI, used for retrieving current web pages for Grok search and answer generation.
Its primary user-agent pattern is xAI-SearchBot; related patterns include xAI-SearchBot/1.0; Grok search fetcher; a representative HTTP user-agent is Mozilla/5.0 (compatible; xAI-SearchBot/1.0; +https://x.ai).
xAI-SearchBot is unverified with Medium confidence. The identity type is Observed.
Robots.txt behavior for xAI-SearchBot is not confirmed. Do not treat a user-agent-only rule as guaranteed enforcement.
The exact pattern has been observed in server logs, but xAI does not currently publish a crawler-verification method for this identity.
Treat the user-agent as an identity claim. Corroborate it with operator-controlled IP ranges, reverse DNS, signed-request evidence, or current documentation when available.
Identity
- User-Agent Pattern
-
xAI-SearchBot - Aliases
- xAI-SearchBot/1.0; Grok search fetcher
- HTTP Agent Examples
-
Mozilla/5.0 (compatible; xAI-SearchBot/1.0; +https://x.ai) - Robots Token
- xAI-SearchBot
- Identity Type
- Observed
- Evidence Method
- Match `xAI-SearchBot` and corroborate the request with current xAI documentation, operator-controlled network data, reverse DNS, or signed-request evidence when available.
Classification
- Type
- AI
- Kind
- Fetcher
- Family
- xAI Grok
- Operator
- xAI
- Company
- xAI
- Purpose
- AI search retrieval
Behavior and handling
- Common Use
- retrieving current web pages for Grok search and answer generation
- Detection Notes
- Detect primarily by `xAI-SearchBot`; related patterns include `xAI-SearchBot/1.0; Grok search fetcher`; representative HTTP user-agent: `Mozilla/5.0 (compatible; xAI-SearchBot/1.0; +https://x.ai)`. User-agent strings can be copied, so corroborate with stronger evidence before granting trust.
- Respects robots.txt
- Unknown
- Spoofing Risk
- xAI-SearchBot has high spoofing risk because its user-agent can be copied and the current evidence is observation-based; do not trust it by name alone.
- Risk
- Neutral
- Recommended Handling
- Monitor
Rules and controls
- Robots.txt Snippet
-
# Robots.txt behavior is unconfirmed. Monitor first and use stronger controls when enforcement is required. - Apache Rule
-
RewriteCond %{HTTP_USER_AGENT} "xAI\-SearchBot" [NC] RewriteRule ^ - [F,L] - Nginx Rule
-
if ($http_user_agent ~* "xAI\-SearchBot") { return 403; }