scraper

AllAfrica Safe

All Africa Crawler

All Africa Crawler is a crawler used for scraping, SEO analysis, or data collection from AllAfrica Global Media.

Operator and family: AllAfrica Global Media Type: Scraper Purpose: Scraping

Verified robots.txt: No Block: Depends Verified Bot

AppcastParsingBot Neutral

AppcastBot

Scrapes client job postings for Appcast services.

Operator and family: Appcast Type: Scraper Purpose: Scraping

Verified robots.txt: No Block: Depends Verified Bot

Attracta Neutral

Attracta

The Attracta bot is analyzes user website content as part of Attracta's SEO services.

Operator and family: Attracta Type: Scraper Purpose: Scraping

Verified robots.txt: No Block: Depends Verified Bot

BigUpDataBot Neutral

BigUpData Bot

BigUpData Bot is a crawler used for scraping, SEO analysis, or data collection from BigUpData.

Operator and family: BigUpData Type: Scraper Purpose: Scraping

Verified robots.txt: No Block: Depends Verified Bot

BingPreview Neutral

Bing Preview

BingPreview generates page snapshots for Bing. Note that BingPreview has desktop and mobile variants.

Operator and family: Microsoft Type: Scraper Purpose: Scraping

Verified robots.txt: No Block: Depends Verified Bot

channable/importer Neutral

Channable

Channable is a crawler used for scraping, SEO analysis, or data collection from Channable.

Operator and family: Channable Type: Scraper Purpose: Scraping

Verified robots.txt: No Block: Depends Verified Bot

coccocbot Safe

Cốc Cốc

Coccocbot scrapes websites that are request from the Vietnamese search engine Coc Coc.

Operator and family: Coccoc Type: Scraper Purpose: Scraping

Verified robots.txt: No Block: Depends Verified Bot

ContextualBot Neutral

ContextualBot

Accesses page content to create contextual segments for targeting within several DPS & SSP platforms.

Operator and family: Outcomes Type: Scraper Purpose: Scraping

Verified robots.txt: No Block: Depends Verified Bot

RSiteAuditor Neutral

DataForSEO

DataForSEO is using RSiteAuditor to scan websites for critical on-site SEO errors and provides aggregated data in a structured form to its customer through a RESTful API.

Operator and family: DataForSEO Type: Scraper Purpose: Scraping

Verified robots.txt: No Block: Depends Verified Bot

DataForSeoBot Neutral

DataForSEO Bot

DataForSEO Bot is a driving force of our leading product - Backlinks API, which has been developed with a single purpose: providing website owners, webmasters, and SEO professionals with.

Operator and family: DataForSEO Type: Scraper Purpose: Scraping

Verified robots.txt: No Block: Depends Verified Bot

Diffbot-User Neutral

Diffbot-User

Diffbot-User is a crawler used for scraping, SEO analysis, or data collection from Diffbot.

Operator and family: Diffbot Type: Scraper Purpose: Scraping

Verified robots.txt: Yes Block: Depends Official Documented

DigiCert DCV Bot Neutral

DigiCert DCV

DigiCert DCV is a crawler used for scraping, SEO analysis, or data collection from DigiCert.

Operator and family: DigiCert Type: Scraper Purpose: Scraping

Verified robots.txt: No Block: Depends Verified Bot

Discordbot Neutral

discordBot

The discordBot scrapes URLs that are shared within the Discord chat platform. This is done to generate contextual previews of the content, including titles, descriptions, and images.

Operator: Discord, Inc. Family: Discord Type: Scraper Purpose: Scraping

Verified robots.txt: No Block: Depends Verified Bot

w4mwnpbXf3MFAbxOkJRw Neutral

Echobot Bot

We scrape full article/page content to ensure we can optimally automate the content distribution for the digital publishers we work with. Every single article a publisher releases will get.

Operator and family: Echobox Type: Scraper Purpose: Scraping

Verified robots.txt: No Block: Depends Verified Bot

FastmailUA Neutral

Fastmail Bot

Fastmail Bot is a crawler or scraper associated with Fastmail, used for public-data collection, crawling, or content extraction.

Operator and family: Fastmail Type: Scraper Purpose: Scraping

Verified robots.txt: No Block: Depends Verified Bot

Fedica Neutral

Fedicabot

Fedicabot is a crawler used for scraping, SEO analysis, or data collection from Fedica.

Operator and family: Fedica Type: Scraper Purpose: Scraping

Verified robots.txt: No Block: Depends Verified Bot

FlipboardProxy Neutral

FlipboardProxy

We work with publishers/partners to obtain their content so it is formatted for the Flipboard app.

Operator and family: Flipboard Type: Scraper Purpose: Scraping

Verified robots.txt: No Block: Depends Verified Bot

Google-Firebase Neutral

Google Firebase

Google Firebase is a crawler used for scraping, SEO analysis, or data collection from Google.

Operator and family: Google Type: Scraper Purpose: Scraping

Verified robots.txt: No Block: Depends Verified Bot

GoogleImageProxy Neutral

Google Image Proxy

The Google Image Proxy bot is used to render link content sent via email in Gmail.

Operator and family: Google Type: Scraper Purpose: Scraping

Verified robots.txt: No Block: Depends Verified Bot

Google-PageRenderer Neutral

Google Page Renderer

Tools and product functions where the end user triggers a fetch.

Operator and family: Google Type: Scraper Purpose: Scraping

Verified robots.txt: No Block: Depends Verified Bot

hey.com/imageproxy Neutral

HEY Email Privacy Proxy

HEY email stops spy pixels and prevents user IP tracking by proxying all HTML email images, fonts, and external assets.

Operator: HEY Family: HEY Email Privacy Proxy Type: Scraper Purpose: Scraping

Verified robots.txt: No Block: Depends Verified Bot

https://iframely.com/docs/about Neutral

Iframely

Iframely is a crawler used for scraping, SEO analysis, or data collection from Iframely.

Operator and family: Iframely Type: Scraper Purpose: Scraping

Verified robots.txt: No Block: Depends Verified Bot

imagesiftbot Safe

ImagesiftBot

ImagesiftBot is a crawler used for scraping, SEO analysis, or data collection from Hive.

Operator and family: Hive Type: Scraper Purpose: Scraping

Verified robots.txt: Unknown Block: Depends Verified Bot

kakaotalk-scrap Neutral

KakaoTalk Scrap

The KakaoTalk scrap server collects and processes webpage data to create optimized previews for URLs.

Operator: Kakao Family: KakaoTalk Scrap Type: Scraper Purpose: Scraping

Verified robots.txt: No Block: Depends Verified Bot