Confidence: Medium
Yext Inc
The Yext Crawler provides Yext customers with a tool to retrieve data from their own websites.
Verified robots.txt: No Neutral Block: Depends
FishBot crawls webpages to deliver Open Source AI for All
User-agent: FishBot
Disallow: /
(http.user_agent contains "FishBot")
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} FishBot [NC]
RewriteRule ^ - [F,L]
if ($http_user_agent ~* "FishBot") { return 403; }
The Yext Crawler provides Yext customers with a tool to retrieve data from their own websites.
A content based scraper only for partners we collaborate with who have given permission to have their website scraped.
Yandex crawler for mobile-layout checks.
Yahoo Mail Proxy is a content fetch proxy that retrieves the page content of URLs that are embedded within emails sent to Yahoo Mail users.
