Botcrawl API

API

Machine-readable bot intelligence for crawlers, AI bots, user agents, and automated traffic.

JSONFeed Format
RESTLookup API
RulesExport Feeds
View Manifest Download Feed Browse Bots

Recommended use: Download the static Botcrawl bot intelligence feed and cache it locally. Software should not call the live lookup endpoint on every pageview or request.

Bot Intelligence for Software, Security Tools, and Website Owners

The Botcrawl API provides machine-readable access to Botcrawl’s bot intelligence feed, including known crawlers, AI bots, preview fetchers, monitors, scrapers, user agents, robots.txt behavior, risk levels, verification notes, and recommended handling.

Use it to identify known bots, support bot monitoring, generate rules, filter automated traffic, research bot operators, or build software that needs structured bot data.

Identify Known Bots

Look up known bots by user-agent and review the operator, purpose, category, confidence, risk level, and verification method.

Use a Static Feed

Download the JSON feed and cache it locally for fast access without querying WordPress on every request.

Generate Rules

Access rule-focused feeds for robots.txt, Apache, Nginx, and Cloudflare-style workflows.

Build Integrations

Use the feed in plugins, monitoring systems, firewall workflows, analytics filters, research tools, and bot management software.

Static Feed Endpoints

The static feed is the recommended way to use Botcrawl bot intelligence in software. It is built for local caching, periodic updates, and fast lookups.

GET /bot-intel/v1/manifest.json
GET /bot-intel/v1/bots.json
GET /bot-intel/v1/bots.min.json
GET /bot-intel/v1/categories.json

Rule Feed Endpoints

Rule feeds separate bot identity from practical handling rules, making them easier to use in blocking, monitoring, robots.txt, and server configuration workflows.

GET /bot-intel/v1/rules/robots.json
GET /bot-intel/v1/rules/apache.json
GET /bot-intel/v1/rules/nginx.json
GET /bot-intel/v1/rules/cloudflare.json

REST API Endpoints

The REST API is useful for lookup, search, testing, documentation, and integration workflows.

GET /wp-json/botcrawl/v1/manifest
GET /wp-json/botcrawl/v1/bots
GET /wp-json/botcrawl/v1/bots/{slug}
GET /wp-json/botcrawl/v1/identify?ua=Googlebot
GET /wp-json/botcrawl/v1/search?q=google
GET /wp-json/botcrawl/v1/categories
GET /wp-json/botcrawl/v1/rules/robots

Identify a Bot by User-Agent

Use the identify endpoint to check whether a user-agent matches a known bot in the Botcrawl directory.

GET /wp-json/botcrawl/v1/identify?ua=YandexVideoParser

Example response:

{
  "matched": true,
  "matches": [
    {
      "id": "yandexvideoparser",
      "name": "YandexVideoParser",
      "operator": "Yandex",
      "category": "search",
      "risk": "low",
      "recommended_action": "depends",
      "confidence": "medium",
      "verified": "yes",
      "respects_robots": "yes"
    }
  ]
}

What Each Bot Record Can Include

Each bot record may include the bot name, slug, user-agent token, operator, company, category, purpose, verification method, robots.txt behavior, risk level, recommended action, source URL, operator URL, and last verified date.

Best Practice

For most software integrations, use the static feed instead of calling the lookup endpoint on every request.

1. Download the manifest.
2. Check the feed version.
3. Download bots.json only when the version changes.
4. Store the feed locally.
5. Match incoming user-agents against the local copy.
6. Verify bots when possible before allow-listing them.

Verification Notice

User-agent strings can be spoofed. A matching user-agent does not always prove that traffic came from the real operator. When possible, verify bots using reverse DNS, forward DNS, official IP ranges, or other operator-documented methods.

Commercial and Advanced Use

Botcrawl may offer advanced access, commercial licensing, signed feeds, higher rate limits, custom exports, historical changes, and integration support for security tools, plugins, analytics platforms, and hosting providers.

For API access questions, commercial licensing, or partnership requests, contact Botcrawl.