Overview
GPTBot is a crawler associated with OpenAI, used for crawling content that may be used to improve and train OpenAI generative AI foundation models.
Its primary user-agent pattern is GPTBot; related patterns include OpenAI training crawler; a representative HTTP user-agent is Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot.
GPTBot is verified with High confidence. The identity type is Official Documented.
GPTBot is documented as honoring robots.txt directives.
Treat the user-agent or identity as a claim and corroborate it with published network data, reverse DNS, signed-request evidence, or current operator documentation when available.
Identity
- User-Agent Pattern
-
GPTBot - Aliases
- OpenAI training crawler
- HTTP Agent Examples
-
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot - Robots Token
- GPTBot
- Identity Type
- Officially documented
- Evidence Method
- Verify GPTBot by matching `GPTBot` to OpenAI evidence, then corroborating the request with source-network ownership, reverse DNS, signed request data, or current operator documentation.
Classification
- Type
- AI
- Kind
- Crawler
- Family
- OpenAI
- Purpose
- AI training
Behavior and handling
- Common Use
- crawling content that may be used to improve and train OpenAI generative AI foundation models
- Detection Notes
- GPTBot traffic is primarily detected by `GPTBot`; related patterns include `OpenAI training crawler`; a representative HTTP user-agent is `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot`. Treat these values as identification claims and corroborate them with source-network ownership, reverse DNS, signed request data, or current operator documentation.
- Respects robots.txt
- Yes
- Spoofing Risk
- GPTBot can be spoofed if identification relies only on a user-agent string; corroborate the request with stronger operator-controlled evidence when available.
- Risk
- Neutral
- Recommended Handling
- Depends
Rules and controls
- Robots.txt Snippet
-
User-agent: GPTBot Disallow: /
Relationships
- Operator
- OpenAI Checked 2026-08-07
Relationships without an Evidence link are normalized from the canonical directory record. They should not be interpreted as independent proof of physical presence or request origin.