Botcrawl is an independent cybersecurity publication and public web intelligence platform founded in 2011. We document automated web activity, internet infrastructure, and online threats through structured public databases and independent reporting.
Botcrawl maintains a searchable Bot Directory covering crawlers, AI agents, scrapers, scanners, monitoring services, and other automated systems. Our Network Directory documents internet networks, autonomous system numbers, providers, and IP ranges that may appear in website traffic and technical investigations.
Alongside these databases, Botcrawl reports on data breaches, malware, scams, ransomware, artificial intelligence, and other security issues affecting people and organizations online.
What Botcrawl Does
Botcrawl helps website owners, researchers, developers, security professionals, journalists, and readers understand the systems operating across the modern web.
Our work focuses on three connected areas:
- Documenting bots, crawlers, automated agents, and their known identifiers
- Organizing public information about networks, providers, ASNs, and IP ranges
- Reporting on cybersecurity incidents, online threats, and important developments affecting the internet
The goal is not to place every automated request or network into a simplistic category. Botcrawl is designed to provide enough context for people to examine the available information, understand its limitations, and make their own informed decisions.
Bot Directory
The Botcrawl Bot Directory is a public reference for researching automated web traffic. It includes records for search crawlers, AI crawlers, user-triggered agents, scrapers, security scanners, monitoring services, feed fetchers, link-preview bots, webhooks, automation tools, and other systems that interact with websites.
Depending on the information available, a bot record may include:
- The bot, crawler, or agent name
- Known user-agent strings and identity patterns
- The operator or organization associated with it
- Its category, purpose, family, or type
- robots.txt tokens and documented controls
- Available verification methods
- Known behavior and practical handling considerations
- The date and confidence level of the available information
Bots serve many different purposes. Some support search indexing, accessibility, uptime monitoring, security testing, link previews, AI-assisted browsing, or user-requested retrieval. Others collect content, probe websites, imitate recognized crawlers, disregard published controls, or generate unwanted automated traffic.
Botcrawl documents those differences rather than treating all automation as inherently safe or inherently malicious.
A user-agent string alone does not prove identity. User agents can be copied, modified, or spoofed. Where appropriate, Botcrawl distinguishes between a claimed identity, official documentation, robots.txt information, network evidence, reverse-DNS verification, cryptographic signals, and observed behavior.
Network Directory
The Botcrawl Network Directory documents internet networks that may appear in server logs, analytics platforms, firewall events, abuse reports, and other technical records.
Network profiles may include:
- Network or provider name
- Autonomous system number, commonly called an ASN
- Registered or commonly associated network name
- Representative IPv4 or IPv6 ranges
- CIDR blocks and example addresses
- Network type or infrastructure classification
- Reverse-DNS and hostname information
- Operating scope and other available technical details
A Network record describes infrastructure associated with a provider, ASN, or IP range. It does not, by itself, identify a physical data center, establish the exact location of a visitor, prove that a request was automated, or show that the traffic was malicious.
Networks can carry many forms of traffic from many unrelated customers and services. Botcrawl presents network information as technical context, not as a verdict about every request originating from an address within that network.
Cybersecurity Reporting
Botcrawl has covered cybersecurity and online threats since 2011. Our reporting documents incidents and developments involving individuals, businesses, public institutions, governments, technology platforms, and internet infrastructure.
Coverage includes:
- Data breaches and exposed information
- Malware campaigns and malicious software
- Scams and online fraud
- Ransomware incidents and cybercrime activity
- Artificial intelligence and automated web systems
- Security issues involving websites, software, networks, and internet services
Cybersecurity stories often develop over time. Early information may come from affected organizations, researchers, regulators, public records, technical evidence, or claims made by attackers. Botcrawl works to distinguish confirmed facts from allegations, estimates, disputed statements, and information that remains incomplete.
When important details cannot be independently confirmed, that uncertainty should remain visible to the reader.
Research and Verification
Botcrawl prioritizes accuracy, context, and clear descriptions of what the available evidence does and does not establish.
Cybersecurity reporting may be informed by public disclosures, regulatory filings, victim notifications, official statements, technical indicators, public records, direct examination of available evidence, and other credible information relevant to an incident.
Bot and Network records may be informed by operator documentation, published technical specifications, user-agent information, robots.txt instructions, public routing and registration data, network ranges, reverse-DNS records, cryptographic identity methods, historical records, and observed automated activity.
Not every record can be verified in the same way. Some operators publish extensive documentation and clear verification procedures. Others disclose only a user agent, a control token, limited network information, or no formal documentation at all.
Botcrawl preserves these distinctions through verification notes, confidence assessments, dates, classifications, and carefully qualified descriptions. A lack of verification does not automatically prove malicious intent, and published documentation does not guarantee that every request claiming an identity is genuine.
Our role is to document, organize, and explain available information. Botcrawl does not participate in extortion, sell stolen data, coordinate with criminal groups, or assist malicious activity.
Editorial Independence
Botcrawl is independently operated. It is not owned by a cybersecurity vendor, hosting provider, artificial intelligence company, advertising network, or infrastructure operator.
This independence allows Botcrawl to report on security incidents, bots, technology companies, networks, and online platforms without changing conclusions to support a vendor or outside organization.
Coverage is based on the available evidence and the public value of the information. The size, influence, or commercial importance of an organization does not determine how Botcrawl describes a documented incident or technical system.
Botcrawl may use advertising, partnerships, or other forms of financial support to operate the website. Such relationships do not determine editorial findings, database classifications, or research conclusions.
Who Botcrawl Is For
Botcrawl is built for people who need understandable and technically responsible information about automated traffic, internet infrastructure, and cybersecurity.
- Website owners investigating bots, crawlers, and unfamiliar traffic
- Security professionals researching threats, infrastructure, and indicators
- Developers examining user agents, ASNs, IP ranges, and automated systems
- Journalists researching cybersecurity incidents and technology companies
- Businesses reviewing traffic sources and online risks
- Researchers studying bots, AI agents, networks, scraping, and web automation
- Readers looking for direct explanations of online threats
Technical accuracy matters, but useful information should still be readable. Botcrawl aims to explain complicated systems without hiding important details behind unnecessary jargon.
Corrections and Contributions
Bot and Network information changes over time. Operators update user agents, infrastructure moves between networks, IP ranges are reassigned, documentation changes, and new automated systems appear regularly.
Botcrawl welcomes corrections, documented updates, bot submissions, Network information, security research, and credible incident leads. Information should include enough detail to identify the affected record or support the proposed correction.
To submit information or contact Botcrawl, visit the Contact page.
Why Botcrawl Exists
The web depends on automated systems and infrastructure that most people rarely see. Search crawlers, AI agents, monitoring services, scrapers, scanners, cloud networks, hosting providers, and malicious automation interact with websites every day.
At the same time, data breaches, malware, fraud, ransomware, and other online threats continue to affect people and organizations around the world.
Botcrawl exists to make this activity easier to identify, research, and understand. Since 2011, that work has continued through independent cybersecurity reporting, the public Bot Directory, and the Network Directory.