Bot Directory

Search the Botcrawl Bot Directory database to identify AI crawlers, search bots, scrapers, scanners, monitoring bots, preview bots, and user-agent patterns.

660 bots
SiteLockSpider Safe

SiteLock

The SiteLock Spider is a web scanning service scans websites for malware and malicious code.

Operator and family: SiteLock Type: Security
Verified robots.txt: No Block: No Verified Bot
SiteSearch360 Safe

SiteSearch360

Site Search 360 is a popular Google Site Search replacement. Our crawler indexes content on our customers' sites for search.

Operator and family: SEMKNOX GmbH Type: Search Purpose: Indexing
Verified robots.txt: No Block: Depends Verified Bot
SiteUptime Safe

SiteUpTimeBot

SiteUpTimeBot is the HTTP monitoring probe for SiteUpTime.com.

Operator and family: SiteUpTime Type: Monitoring
Verified robots.txt: No Block: No Verified Bot
Skroutz ImageBot Neutral

Skroutz ImageBot

Skroutz ImageBot to fetch the individual product images.

Operator and family: Skroutz S.A. Type: Scraper Purpose: Scraping
Verified robots.txt: No Block: Depends Verified Bot
SkroutzBot Safe

SkroutzBot

Skroutz uses SkroutzBot web crawler to download XML feeds.

Operator and family: Skroutz S.A. Type: Feed Purpose: Feed Fetch
Verified robots.txt: No Block: Depends Verified Bot
SkypeUriPreview Neutral

Skype

Skype's URI Preview services fetches a page preview when someone posts a URL in a Skype message.

Operator and family: Skype Type: Scraper Purpose: Scraping
Verified robots.txt: No Block: Depends Verified Bot
Slack-ImgProxy Neutral

Slack Image Proxy

This robot is used to fetch and cache images posted into Slack channels.

Operator and family: Slack Type: Scraper Purpose: Scraping
Verified robots.txt: No Block: Depends Verified Bot
Slackbot Safe

Slackbot

Slackbot is a webhook callback service from Slack used for service notifications, callbacks, or integration events.

Operator and family: Slack Type: Webhook
Verified robots.txt: Yes Block: Depends Official Documented
Slackbot-LinkExpanding Safe

Slackbot-LinkExpanding

Slackbot-LinkExpanding is a preview or prefetch bot from Slack used to fetch pages for previews, unfurls, or user-triggered rendering.

Operator and family: Slack Type: Preview
Verified robots.txt: Yes Block: Depends Official Documented
SlickBot Neutral

Slickstream

Slickstream is a SaaS that indexes our customer's websites (with their approval) in order to provide engagement features for their site visitors, including site search, content.

Operator and family: Slickstream Type: Search Purpose: Indexing
Verified robots.txt: No Block: Depends Verified Bot
slopsearch Neutral

slopsearch

Research web crawler that indexes publicly available documents. Respects robots.txt and rate limits.

Operator: Christopher Albert Family: Christopher Type: Search Purpose: Indexing
Verified robots.txt: No Block: Depends Verified Bot
SmarshBot Safe

SmarshBot

SmarshBot is a security scanner from https://central.smarsh.com/s/ used for security checks, malware review, or abuse prevention.

Operator and family: https://central.smarsh.com/s/ Type: Security
Verified robots.txt: No Block: No Verified Bot
SmartologyBot Neutral

SmartologyBot

The Smartology generates semantic vectors from domain pages in order to serve semantically-relevant ads on those pages.

Operator and family: Smartology Type: Scraper Purpose: Scraping
Verified robots.txt: No Block: Depends Verified Bot
SMTnetPMBot Safe

SMTnet PM Bot

Crawls partner company's websites to include them in our on-site search engine.

Operator and family: SMTnet Type: Search Purpose: Indexing
Verified robots.txt: Yes Block: Depends Verified Bot
Snipcart/1.0 Safe

Snipcart

Snipcart is an e-commerce solution for developers.

Operator and family: Snipcart Type: Ecommerce
Verified robots.txt: No Block: No Verified Bot
Sogou web spider Neutral

Sogou web spider

Sogou crawler identifier used for Sogou search and related Chinese search surfaces.

Operator and family: Sogou Type: Search Purpose: Search Crawling
Not verified robots.txt: Unknown Block: Depends Observed
solarwinds Safe

SolarWinds Observability

Default header that identifies SolarWinds Observability robots that are used for our synthetic monitoring.

Operator and family: SolarWinds Type: Monitoring
Verified robots.txt: No Block: No Verified Bot
Sora POS Neutral

Sora Caisse POS

Sora Caisse POS is a commerce-related fetcher from Sora Caisse used for commerce, checkout, advertising, or payment-related checks.

Operator: Sora Caisse Family: Sora Caisse POS Type: Ecommerce
Verified robots.txt: No Block: Depends Verified Bot
SparkShipping Safe

Spark Shipping

Spark Shipping is eCommerce automation software for retailers running WooCommerce.

Operator and family: Spark Shipping Type: Ecommerce
Verified robots.txt: No Block: No Verified Bot
SparkPost Safe

SparkpostBot

Sparkbot webhook integration is used for automating email transactions on web server events.

Operator and family: Bird Type: Webhook
Verified robots.txt: No Block: No Verified Bot
Spectate Safe

Spectate

Spectate is a monitoring bot associated with Spectate, used for uptime monitoring, site checks, and managed website testing.

Operator and family: Spectate Type: Monitoring
Verified robots.txt: No Block: No Verified Bot
SplitSignalBot Safe

SplitSignalBot

SplitSignalBot is a search crawler from Semrush used for search indexing, page discovery, or content rendering.

Operator and family: Semrush Type: Search Purpose: Indexing
Verified robots.txt: Yes Block: Depends Official Documented
Rigor Safe

Splunk

Synthetics monitoring platform used by Enterprise organizations.

Operator and family: Splunk Type: Monitoring
Verified robots.txt: No Block: No Verified Bot
TwinWaveScanner Safe

Splunk Attack Analyzer

Splunk Attack Analyzer (formerly known as TwinWave), visits URLs submitted by customers using a headless Chrome browser. DOM (Document Object Model), HAR (HTTP Archive), and other relevant.

Operator and family: Splunk Type: Feed Purpose: Ai Training
Verified robots.txt: No Block: No Verified Bot