Overview
DigitalOcean Gradient AI Crawler is DigitalOcean’s documented website crawler for indexing website content into Gradient AI knowledge bases.
DigitalOcean documents the crawler as DigitalOceanGradientAICrawler/1.0. It follows site links according to the selected knowledge-base crawl behavior and respects robots.txt allow and disallow directives.
The former Botcrawl pattern DigitalOceanGenAI-Crawler is retained only as a historical alias for detection compatibility; it is not the current documented identity.
Identity
- User-Agent Pattern
-
DigitalOceanGradientAICrawler - Aliases
- DigitalOceanGradientAICrawler/1.0; DigitalOcean Gradient AI Crawler; DigitalOcean GenAI Crawler; DigitalOceanGenAI-Crawler
- HTTP Agent Examples
-
DigitalOceanGradientAICrawler/1.0 - Robots Token
- DigitalOceanGradientAICrawler
- Identity Type
- Officially documented
- Evidence Method
- Match `DigitalOceanGradientAICrawler` in the User-Agent and corroborate the behavior against DigitalOcean’s current crawler documentation. A User-Agent string alone can be spoofed.
Classification
- Type
- AI
- Kind
- Crawler
- Family
- DigitalOcean
- Purpose
- Knowledge base indexing
Behavior and handling
- Common Use
- Indexing website content supplied as a data source for DigitalOcean Gradient AI knowledge bases.
- Detection Notes
- Current documented traffic uses `DigitalOceanGradientAICrawler/1.0`. Treat `DigitalOceanGenAI-Crawler` only as a historical/legacy detection alias unless independently observed and verified.
- Respects robots.txt
- Yes
- Spoofing Risk
- The User-Agent can be copied; verify surrounding request behavior and current operator documentation before trusting the identity.
- Risk
- Neutral
- Recommended Handling
- Depends
Rules and controls
- Robots.txt Snippet
-
User-agent: DigitalOceanGradientAICrawler Disallow: /
Relationships
- Operator
- DigitalOcean Checked 2026-08-07
Relationships without an Evidence link are normalized from the canonical directory record. They should not be interpreted as independent proof of physical presence or request origin.