ProjectShield Url Check
Project Shield, created by Google Cloud and Jigsaw and powered by Google Cloud Armor, provides free unlimited protection against DDoS attacks, a type of digital attackā¦
Google Ads mobile crawler used to check mobile landing-page quality.
User-agent: AdsBot-Google-Mobile
Disallow: /
(http.user_agent contains "AdsBot-Google-Mobile")
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} AdsBot\-Google\-Mobile [NC]
RewriteRule ^ - [F,L]
if ($http_user_agent ~* "AdsBot-Google-Mobile") { return 403; }
AdsBot-Google-Mobile is Google’s mobile Google Ads crawler for ad landing-page quality checks. Google documents that it ignores the global robots.txt wildcard with publisher permission and should be controlled with its own token.
https://developers.google.com/crawling/docs/crawlers-fetchers/google-special-case-crawlers
Project Shield, created by Google Cloud and Jigsaw and powered by Google Cloud Armor, provides free unlimited protection against DDoS attacks, a type of digital attackā¦
Generic Google crawler for public-content fetches outside core Search.
GoogleOther-Image is GoogleOther optimized for fetching publicly accessible image URLs.
GoogleOther-Video is GoogleOther optimized for fetching publicly accessible video URLs.
Google Messages link-preview fetcher.
NotebookLM fetcher for user-supplied sources.
Google's robots token for Google News crawling preferences.
Google's main crawler for Search and related surfaces.
FishBot crawls webpages to deliver Open Source AI for All
The Yext Crawler provides Yext customers with a tool to retrieve data from their own websites.
A content based scraper only for partners we collaborate with who have given permission to have their website scraped.
Yandex crawler for mobile-layout checks.
