Botcrawl is an independently operated cybersecurity publication and public directory project founded in 2011.
Botcrawl maintains structured public information about identifiable web bots, internet networks, and physical data centers and related infrastructure, alongside reporting on cybersecurity, online threats, artificial intelligence, and internet infrastructure.
The website is free to use. Its directories are designed to make technical information easier to find, compare, and understand while keeping software identity, routing context, and physical infrastructure clearly separated.
The directories describe different layers of internet activity and infrastructure: Bots identify automated clients, Networks describe routing infrastructure, and Data Centers describe physical sites. The Data Center Map provides a geographic view of mapped directory records.
Identifies automated web clients and agents, including crawlers, AI agents, scrapers, scanners, monitoring services, preview bots, and user-triggered agents.
A Bot record describes the automated client itself. It can document User-Agent strings, robots.txt tokens, operators, purposes, verification methods, and observed behavior without treating the Network carrying its traffic as the Bot.
Documents autonomous systems (ASNs) and their routing context, including carriers, access providers, hosting networks, cloud platforms, CDNs, DNS services, and other infrastructure providers.
Each Network record is anchored to an ASN and documents supported routing and provider context. Network information can help explain where traffic is routed, but it does not by itself identify the Bot that made a request or the physical site that handled it.
Documents physical data centers and related qualifying infrastructure sites, including colocation facilities, carrier hotels, interconnection facilities, edge sites, and other locations within the directory scope.
Each record describes a physical site and identifies its facility type, operator, location, status, and other supported infrastructure details when that information is available.
Provides a geographic view of mapped Data Center Directory records so physical infrastructure can be explored by location instead of one profile at a time.
Use the map to search, filter, inspect locations, and open the underlying directory records. Map positions reflect the location information and precision available for each record and should not always be interpreted as exact building coordinates.
Botcrawl uses these terms consistently so software, routing infrastructure, physical infrastructure, and supporting evidence are not conflated. They describe how this project organizes its records.
Directory records are maintained as structured research records. Sources, uncertainty, corrections, and record relationships are handled separately so a single observation is not treated as proof of more than it establishes.
Records can draw from operator documentation, technical documentation, routing information, public records, credible reporting, direct technical observations, and sources submitted by readers or organizations. Different fields within one record may rely on different evidence. Public evidence identifies its source authority and review freshness where available; source authority describes provenance and does not automatically make every associated claim certain.
Every contribution is manually reviewed before it can change a public directory record. Botcrawl checks whether it is within scope, compares it with existing records, reviews the supporting sources, and applies only information that can be supported.
Verification, evidence confidence, source authority, freshness, status, dates, and location precision describe specific parts of a record. A well-supported identity does not automatically verify every attributed behavior, relationship, address, or historical detail, and estimates are kept distinct from confirmed information where possible. The Changes timeline records selected transitions only after Botcrawl establishes its history baseline; it does not invent earlier history.
Inclusion means the subject falls within the directory scope and there is sufficient information to document it. It does not imply endorsement, partnership, certification, safety, maliciousness, ownership, or responsibility for activity associated with the record.