Overview
DigitalOcean Gradient AI Crawler is DigitalOcean’s documented website crawler for indexing website content into Gradient AI knowledge bases.
DigitalOcean documents the crawler as DigitalOceanGradientAICrawler/1.0. It follows site links according to the selected knowledge-base crawl behavior and respects robots.txt allow and disallow directives.
The former Botcrawl pattern DigitalOceanGenAI-Crawler is retained only as a historical alias for detection compatibility; it is not the current documented identity.
Identity
- User-Agent
DigitalOceanGradientAICrawler- Aliases
- DigitalOceanGradientAICrawler/1.0; DigitalOcean Gradient AI Crawler; DigitalOcean GenAI Crawler; DigitalOceanGenAI-Crawler
- HTTP Agent Examples
DigitalOceanGradientAICrawler/1.0- Robots.txt Token
DigitalOceanGradientAICrawler- Identity Type
- Official Documented
- Evidence Method
- Match `DigitalOceanGradientAICrawler` in the User-Agent and corroborate the behavior against DigitalOcean’s current crawler documentation. A User-Agent string alone can be spoofed.
Classification
- Type
- AI
- Kind
- Crawler
- Family
- DigitalOcean
- Purpose
- knowledge-base-indexing
Behavior and handling
- Common Use
- Indexing website content supplied as a data source for DigitalOcean Gradient AI knowledge bases.
- Detection Notes
- Current documented traffic uses `DigitalOceanGradientAICrawler/1.0`. Treat `DigitalOceanGenAI-Crawler` only as a historical/legacy detection alias unless independently observed and verified.
- Respects robots.txt
- Yes
- Spoofing Risk
- The User-Agent can be copied; verify surrounding request behavior and current operator documentation before trusting the identity.
- Risk
- Neutral
- Recommended Handling
- Depends
Rules and controls
- Robots.txt Snippet
User-agent: DigitalOceanGradientAICrawler Disallow: /