Bot Directory

Search the Botcrawl Bot Directory database to identify AI crawlers, search bots, scrapers, scanners, monitoring bots, preview bots, and user-agent patterns.

2checkout Safe

2checkout

2Checkout Payment Platform's webhooks integration for notifying websites about payment events. Now operated by Verifone.

Operator and family: 2checkout Type: Webhook
Verified robots.txt: No Block: No Verified Bot
monitoring360 Safe

360Monitoring

360Monitoring is a monitoring bot from 360Monitoring used to check website availability, performance, or page health.

Operator and family: 360Monitoring Type: Monitoring
Not verified robots.txt: No Block: No Observed
AccessibleWebBot Safe

Accessible Web Bot

Accessible Web Bot crawls customer websites to discover pages and monitor for accessibility violations on a regular basis. Crawls are initiated for Accessible Web's "Page Monitoring" SaaS.

Operator and family: Accessible Web Type: Monitoring
Verified robots.txt: No Block: No Verified Bot
AccessStatus Safe

AccessStatus

AccessStatus is a bot that checks the HTTP status code of a web page. It is used to determine if a URL is active, redirected, or returning an error.

Operator and family: AccessStatus Type: Monitoring
Verified robots.txt: No Block: No Verified Bot
ActiveComply Safe

ActiveComply Bot

ActiveComply Bot is a crawler for a service that monitors social media for regulatory compliance. It scans for specific keywords and content for businesses in regulated industries.

Operator and family: ActiveComply Type: Security
Verified robots.txt: No Block: No Verified Bot
Adagiobot Safe

Adagio Bot

Adagio Bot is a monitoring bot from Adagio used to check website availability, performance, or page health.

Operator and family: Adagio Type: Monitoring
Verified robots.txt: No Block: No Verified Bot
AddSearchBot Neutral

AddSearchBot

As AddSearch adds content from your site to the search, the AddSearch bot gets counted as traffic by most analytics software.

Operator and family: Addsearch Type: Search Purpose: Indexing
Verified robots.txt: No Block: Depends Verified Bot
AddThis Neutral

AddThis

The AddThis bot crawls websites to gather and update content for its website marketing tools. These tools include features like social sharing buttons and content recommendation widgets.

Operator and family: AddThis Type: Search Purpose: Indexing
Verified robots.txt: No Block: Depends Verified Bot
AdsBot-Google-Mobile Safe

AdsBot-Google-Mobile

AdsBot-Google-Mobile is a search crawler from Google used for search indexing, page discovery, or content rendering.

Operator and family: Google Type: Search Purpose: Indexing
Verified robots.txt: No Block: Depends Official Documented
adsnaver Safe

adsnaver

adsnaver is a search crawler from Naver used for search indexing, page discovery, or content rendering.

Operator and family: Naver Type: Search Purpose: Indexing
Verified robots.txt: No Block: Depends Verified Bot
InteractiveAdvertisingBureau Neutral

AdsTxtCrawler

OneTag is required to crawl ads.txt files from our publishers to verify the presence of the onetag.com line.

Operator and family: OneTag Limited Type: Search Purpose: Indexing
Verified robots.txt: No Block: Depends Verified Bot
Advailo Kuma Neutral

Advailo Kuma

The Advailo Kuma uptime monitor is used to monitor webpage availability and performance.

Operator and family: Advailo Type: Monitoring
Verified robots.txt: No Block: Depends Verified Bot
Adyen Safe

Adyen

The Adyen webhooks integration sends HTTP requests to inform web servers about payment-related events.

Operator and family: Adyen Type: Webhook
Verified robots.txt: No Block: No Verified Bot
AfterShip Magento-2 Connector Safe

AfterShip Magento 2 Connector

AfterShip Magento 2 Connector is a commerce-related fetcher from AfterShip used for commerce, checkout, advertising, or payment-related checks.

Operator and family: AfterShip Type: Ecommerce
Verified robots.txt: No Block: No Verified Bot
aa/1.0 Neutral

Agency Analytics Crawler

A web crawler by Agency Analytics that allows their clients to check their own sites for SEO.

Operator and family: Agency Analytics Type: Search Purpose: Indexing
Verified robots.txt: No Block: Depends Verified Bot
AgencyToolsPro Neutral

AgencyToolsPro Uptime Monitoring Bot

Website monitoring bot, let's you know that your website is down.

Operator and family: AgencyToolsPro Type: Monitoring
Verified robots.txt: No Block: Depends Verified Bot
AhrefsSiteAudit Neutral

Ahrefs Site Audit

Ahrefs Site Audit is a search crawler from Ahrefs used for search indexing, page discovery, or content rendering.

Operator and family: Ahrefs Type: Search Purpose: Indexing
Verified robots.txt: No Block: Depends Verified Bot
AhrefsBot Neutral

AhrefsBot

AhrefsBot is a Web Crawler that powers the 12 trillion link database for Ahrefs online marketing toolset. It constantly crawls web to fill our database with new links and check the status.

Operator and family: Ahrefs Type: Search Purpose: Indexing
Verified robots.txt: No Block: Depends Verified Bot
AI2Bot Neutral

AI2Bot

Allen Institute for AI crawler used for AI research and related indexing workflows.

Operator and family: Allen Institute for AI Type: Ai Purpose: AI Research Crawling
Verified robots.txt: Yes Block: Depends Official Documented
Akismet Safe

Akismet

The integration API for the Akismet spam filtering service.

Operator and family: Akismet Type: Security
Verified robots.txt: No Block: No Verified Bot
AlertSite Safe

Alertsite by Smartbear

Alertsite by Smartbear is the HTTP monitoring probe that monitors its customers websites for availability and performance anomalies.

Operator and family: Smartbear Type: Monitoring
Verified robots.txt: No Block: No Verified Bot
Algolia Crawler Safe

Algolia

Algolia is a search crawler from Algolia used for search indexing, page discovery, or content rendering.

Operator and family: Algolia Type: Search Purpose: Indexing
Verified robots.txt: No Block: Depends Verified Bot
AllAfrica Safe

All Africa Crawler

All Africa Crawler is a crawler used for scraping, SEO analysis, or data collection from AllAfrica Global Media.

Operator and family: AllAfrica Global Media Type: Scraper Purpose: Scraping
Verified robots.txt: No Block: Depends Verified Bot
AmazonAdBot Neutral

Amazon AdBot

Amazon AdBot is a crawler used by Amazon's advertising services. It visits advertiser landing pages to ensure they are compliant with advertising policies.

Operator and family: Amazon Type: Search Purpose: Indexing
Verified robots.txt: No Block: Depends Verified Bot