Bot Directory

Search the Botcrawl Bot Directory database to identify AI crawlers, search bots, scrapers, scanners, monitoring bots, preview bots, and user-agent patterns.

660 bots
rogerbot-crawler Neutral

Moz rogerbot

Rogerbot is Moz's site audit crawler for Moz Pro Campaigns.

Operator and family: Moz Type: Search Purpose: Indexing
Verified robots.txt: Yes Block: Depends Verified Bot
MRGbot Safe

MRGbot

MRGbot is a crawler used for scraping, SEO analysis, or data collection from MRG Web Services srl.

Operator and family: MRG Web Services srl Type: Scraper Purpose: Scraping
Verified robots.txt: Yes Block: Depends Verified Bot
msnbot Safe

MSN

MSNBot was the web crawler for Microsoft's MSN Search, which has since been replaced by Bing. Its purpose was to index web pages for inclusion in the MSN search engine.

Operator and family: Microsoft Type: Search Purpose: Indexing
Verified robots.txt: No Block: Depends Verified Bot
muckrack.com Safe

Muck Rack Scraper

Muck Rack uses many approaches for source discovery such as RSS feeds, sitemaps, and other structured formats.

Operator and family: Muck Rack Type: Feed Purpose: Feed Fetch
Verified robots.txt: No Block: Depends Verified Bot
Nava/ Neutral

Nava Labs ASP (Dev)

Bot to assist social workers in navigating safety net benefit websites.

Operator: Nava Labs Family: Nava Type: Ai Purpose: Ai Assistant
Verified robots.txt: No Block: Depends Verified Bot
naver-blueno Safe

naver-blueno

Naver-blueno is a search crawler associated with Naver, used for search indexing or content discovery.

Operator and family: Naver Type: Search Purpose: Indexing
Verified robots.txt: Unknown Block: Depends Verified Bot
naverbot Safe

naverbot

naverbot is a search crawler from Naver used for search indexing, page discovery, or content rendering.

Operator and family: Naver Type: Search Purpose: Indexing
Verified robots.txt: Unknown Block: Depends Verified Bot
Neevabot Safe

Neevabot

Neevabot is the web crawler for the search engine neeva.com.

Operator and family: Neeva Type: Search Purpose: Indexing
Verified robots.txt: No Block: Depends Verified Bot
Netcraft Safe

Netcraft

The Netcraft Survey Agent is a bot that analyzes web server technology stacks for their Web Server Survey.

Operator and family: Netcraft Type: Security
Verified robots.txt: No Block: No Verified Bot
netEstate NE Crawler Neutral

netEstate Imprint Crawler

The NetEstate Imprint crawler crawls websites for public contact information.

Operator and family: netEstate Type: Search Purpose: Indexing
Verified robots.txt: No Block: Depends Verified Bot
netumo Safe

Netumo

Netumo is a monitoring bot from Netumo used to check website availability, performance, or page health.

Operator and family: Netumo Type: Monitoring
Verified robots.txt: No Block: No Verified Bot
NETVIGIE Safe

NETVIGIE

We checks availability and performances of our client's websites and mobile apps.

Operator and family: Netvigie Type: Monitoring
Verified robots.txt: No Block: No Verified Bot
NewRelicbot Safe

New Relic

The New Relic bot is used by New Relic's Intelligent Observability Platform to monitor customer applications for availability and performance issues.

Operator and family: New Relic Type: Monitoring
Verified robots.txt: No Block: No Verified Bot
scraping@nytimes.com Neutral

New York Times Newsgathering

New York Times Newsgathering is a crawler used for scraping, SEO analysis, or data collection from The New York Times.

Operator and family: The New York Times Type: Scraper Purpose: Scraping
Verified robots.txt: No Block: Depends Verified Bot
NewsBank Neutral

NewsBank

NewsBank aggregates licensed publisher content for schools, libraries, and government research, learning, and archiving.

Operator and family: NewsBank Type: Search Purpose: Indexing
Verified robots.txt: No Block: Depends Verified Bot
NewsBlur Safe

NewsBlur

NewsBlur is a feed retrieval bot from NewsBlur used to retrieve feeds, subscriptions, podcasts, or syndicated content.

Operator and family: NewsBlur Type: Feed Purpose: Feed Fetch
Verified robots.txt: No Block: Depends Verified Bot
NewsNow Safe

NewsNow

The NewsNow bot is the web crawler for the news aggregator service NewsNow.

Operator and family: NewsNowUK Type: Search Purpose: Indexing
Verified robots.txt: No Block: Depends Verified Bot
Nitro- Neutral

NitroBot

We run a cloud based site speed optimization solution. As such, we need to make requests to our clients' sites in order to fetch the content that needs to be optimized. We have several sub.

Operator and family: NitroPack Ltd Type: Webhook
Verified robots.txt: No Block: Depends Verified Bot
NIXStatsbot/1.1 Safe

NixStatsMonitoringBot

The NixStatsMonitoringBot is the HTTP monitoring probe for NixStats to monitor website availability and performance.

Operator and family: NixStats Type: Monitoring
Verified robots.txt: No Block: No Verified Bot
NodePing Safe

Nodeping

The NodePing HTTP Monitoring probe monitors customer websites for uptime.

Operator and family: Nodeping Type: Monitoring
Verified robots.txt: No Block: No Verified Bot
Noibu Safe

Noibu JS Beautifier

Noibu JS Beautifier is a monitoring bot from Noibu used to check website availability, performance, or page health.

Operator and family: Noibu Type: Monitoring
Verified robots.txt: No Block: No Verified Bot
noorobot Neutral

Noorobot

Noorobot is an SEO tool that periodically crawls customer websites to provide recommendations and identify potential SEO-impacting problems.

Operator and family: Noor Digital Agency AB Type: Search Purpose: Indexing
Verified robots.txt: No Block: Depends Verified Bot
Nooshub Safe

Nooshub

Nooshub is a feed retrieval bot from Nooshub used to retrieve feeds, subscriptions, podcasts, or syndicated content.

Operator and family: Nooshub Type: Feed Purpose: Feed Fetch
Verified robots.txt: No Block: Depends Verified Bot
NostoCrawlerBot Neutral

NostoCrawlerBot

NostoCrawlerBot is a search crawler from Nosto used for search indexing, page discovery, or content rendering.

Operator and family: Nosto Type: Search Purpose: Indexing
Verified robots.txt: No Block: Depends Verified Bot