About this bot
DigitalOcean Gradient AI Crawler is associated with DigitalOcean and used for knowledge base indexing.
The primary identifying User-Agent token is DigitalOceanGradientAICrawler. Representative HTTP request identity: DigitalOceanGradientAICrawler/1.0.
Directory status: Active. Identity classification: Verified with High confidence.
robots.txt behavior: yes under DigitalOceanGradientAICrawler.
Indexes website text and supported images into a DigitalOcean knowledge base for retrieval-augmented AI workflows.
How to identify this bot
- User-agent and crawler identifiers
-
DigitalOceanGradientAICrawlerDigitalOceanGradientAICrawler/1.0 - Also known as
- DigitalOceanGradientAICrawler/1.0; DigitalOcean GenAI Crawler
Details
- Kind
- Knowledge base crawler
- Family
- DigitalOcean Gradient AI
- Purpose
- Knowledge base indexing
- Common Use
- Indexes website text and supported images into a DigitalOcean knowledge base for retrieval-augmented AI workflows.
- Respects robots.txt
- Yes
Verification
- Identity Type
- Officially documented
- Evidence Method
- Match the exact `DigitalOceanGradientAICrawler/1.0` identity and corroborate with expected DigitalOcean knowledge-base ingestion activity.
- Detection Notes
- DigitalOcean uses this crawler only when a website or sitemap is configured as a Gradient/Inference knowledge-base source.
Website controls
- Robots.txt Snippet
User-agent: DigitalOceanGradientAICrawler Disallow: /- Apache Rule
RewriteCond %{HTTP_USER_AGENT} "DigitalOceanGradientAICrawler" [NC] RewriteRule ^ - [F,L]- Nginx Rule
if ($http_user_agent ~* "DigitalOceanGradientAICrawler") { return 403; }