Bot Directory

HubSpot Page Fetcher Neutral

HubSpot Page Fetcher

When posting to LinkedIn from Hubspot, images need to be pulled through to LinkedIn when published. The crawler performs this function.

Operator and family: HubSpot Type: Search Purpose: Indexing

Verified robots.txt: No Block: Depends Verified Bot

Huckabot Neutral

Huckabuy Bot

Huckabot is Huckabuy’s main crawler which is utilized by almost all of Huckabuy’s products. The primary purpose of Huckabot is to crawl and index a customer’s website, which is then.

Operator and family: Huckabuy Type: Search Purpose: Indexing

Verified robots.txt: No Block: Depends Verified Bot

hydrozen Safe

Hydrozen

Hydrozen is a monitoring bot from Hydrozen used to check website availability, performance, or page health.

Operator and family: Hydrozen Type: Monitoring

Verified robots.txt: Unknown Block: Depends Verified Bot

Hype Machine Safe

Hype Machine

Since 2005, Hype Machine monitors music publications/blogs for posts about new artists and builds playlists using this metadata for listeners.

Operator and family: Hype Machine Type: Monitoring

Verified robots.txt: No Block: Depends Verified Bot

ias_crawler Neutral

IAS crawler

IAS crawler is a security scanner from Integral Ad Science used for security checks, malware review, or abuse prevention.

Operator and family: Integral Ad Science Type: Security

Verified robots.txt: No Block: Depends Verified Bot

IbouBot Safe

IbouBot

IbouBot is the crawler of the Ibou Search Engine.

Operator and family: Babbar Type: Search Purpose: Indexing

Verified robots.txt: No Block: Depends Verified Bot

ICC-Crawler Neutral

ICC Crawler

ICC-Crawler automatically crawls the Internet and collects web pages. ICC-Crawler is operated by the Universal Communication Research Institute at the National Institute of Information and.

Operator and family: NICT Type: Search Purpose: Indexing

Verified robots.txt: Yes Block: Depends Verified Bot

https://iframely.com/docs/about Neutral

Iframely

Iframely is a crawler used for scraping, SEO analysis, or data collection from Iframely.

Operator and family: Iframely Type: Scraper Purpose: Scraping

Verified robots.txt: No Block: Depends Verified Bot

IFTTT Safe

IFTTT RSS Feed Service

RSS feed fetcher to power user-configured automations.

Operator and family: IFTTT RSS Feed Servuce Type: Feed Purpose: Feed Fetch

Verified robots.txt: No Block: Depends Verified Bot

imagesiftbot Safe

ImagesiftBot

ImagesiftBot is a crawler used for scraping, SEO analysis, or data collection from Hive.

Operator and family: Hive Type: Scraper Purpose: Scraping

Verified robots.txt: Unknown Block: Depends Verified Bot

IndeedJobBot Neutral

IndeedJobBot

Indeed's job crawling bot that crawls job and job related information.

Operator and family: Indeed Type: Search Purpose: Indexing

Verified robots.txt: Yes Block: Depends Verified Bot

inngest Safe

Inngest

Delivers webhook-style event notifications for event-driven applications.

Operator and family: Inngest Type: Webhook

Verified robots.txt: No Block: No Verified Bot

Innguma Safe

Innguma Fetcher

Innguma fetcher collects and periodically refreshes these user-initiated feeds.

Operator and family: Innguma Type: Feed Purpose: Feed Fetch

Verified robots.txt: No Block: Depends Verified Bot

inoreader.com Safe

Innologica

Innologica is a feed retrieval bot from Innologica used to retrieve feeds, subscriptions, podcasts, or syndicated content.

Operator and family: Innologica Type: Feed Purpose: Feed Fetch

Verified robots.txt: No Block: Depends Verified Bot

Instapaper Neutral

Instapaper

Instapaper is an app that lets people save articles to read later.

Operator and family: Instant Paper Type: Feed Purpose: Feed Fetch

Verified robots.txt: No Block: Depends Verified Bot

Integromat Safe

Integromat

Integromat is a webhook callback service from Make used for service notifications, callbacks, or integration events.

Operator and family: Make Type: Webhook

Verified robots.txt: No Block: No Verified Bot

archive.org_bot Neutral

Internet Archive

The Internet Archive bot, also known as archive.org_bot, is the web crawler for the Internet Archive's Wayback Machine. It systematically crawls and preserves publicly accessible web pages.

Operator and family: Internet Archive Type: Feed Purpose: Feed Fetch

Verified robots.txt: Yes Block: Depends Verified Bot

Archive-It Neutral

Internet Archive – Archive-It

Internet Archive’s Archive-It service preserves publicly accessible web pages for the historical record.

Operator and family: Archive-It Type: Feed Purpose: Feed Fetch

Verified robots.txt: No Block: Depends Verified Bot

IABot Safe

InternetArchiveBot

InternetArchiveBot looks for URL references on Wikipedia and assesses if the URL is still alive, or delivering 404s.

Operator and family: Internet Archive Type: Feed Purpose: Feed Fetch

Verified robots.txt: No Block: No Verified Bot

IsDownBot Safe

IsDownBot

IsDown monitors endpoints (websites, APIs) to make sure they are up and running.

Operator and family: IsDownBot Type: Monitoring

Verified robots.txt: No Block: No Verified Bot

www.getuptime.co Safe

Jagged Pixel UptimeBot

Uptime is a synthetic monitoring tool allowing Shopify merchants to validate key customer flows are not broken after making theme changes.

Operator and family: Jagged Pixel Inc. Type: Monitoring

Verified robots.txt: No Block: No Verified Bot

jetmon Safe

Jetpack

Uptime monitor for users of WordPress.com/Jetpack — https://jetpack.com/support/monitor/.

Operator and family: Automattic Type: Monitoring

Verified robots.txt: No Block: No Verified Bot

jobswithgptcom Safe

jobswithgptcom-bot

Simple crawler focussing on only job postings for job search site.

Operator and family: jobswithgptcom Type: Search Purpose: Indexing

Verified robots.txt: No Block: Depends Verified Bot

Jumio Safe

Jumio

Jumio is a security scanner from Coinbase used for security checks, malware review, or abuse prevention.

Operator and family: Coinbase Type: Security

Verified robots.txt: No Block: No Verified Bot