ProjectShield Url Check
Project Shield, created by Google Cloud and Jigsaw and powered by Google Cloud Armor, provides free unlimited protection against DDoS attacks, a type of digital attackā¦
Google feed fetcher used for RSS and Atom retrieval for Google News and WebSub.
Serve a 404 or 410 response to FeedFetcher-Google requests if you need to stop feed retrieval.
(http.user_agent contains "FeedFetcher-Google")
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} FeedFetcher\-Google [NC]
RewriteRule ^ - [F,L]
if ($http_user_agent ~* "FeedFetcher-Google") { return 403; }
FeedFetcher-Google is Google’s feed fetcher for RSS and Atom feeds used by Google News and WebSub. Google says it behaves as a direct agent of the human user and generally ignores robots.txt, recommending server-side error responses if you want to stop it.
https://developers.google.com/crawling/docs/crawlers-fetchers/feedfetcher
Project Shield, created by Google Cloud and Jigsaw and powered by Google Cloud Armor, provides free unlimited protection against DDoS attacks, a type of digital attackā¦
GoogleOther-Image is GoogleOther optimized for fetching publicly accessible image URLs.
GoogleOther-Video is GoogleOther optimized for fetching publicly accessible video URLs.
Google Messages link-preview fetcher.
Generic Google crawler for public-content fetches outside core Search.
Google's main crawler for Search and related surfaces.
Google Pinpoint fetcher for user-supplied sources.
Google control token for Gemini training and grounding permissions.
Research web crawler that indexes publicly available documents. Respects robots.txt and rate limits.
Spyglasses Bot evaluates website content to identify issues that would affect visibility by AI assistants.
AI assistant for e-commerce stores. Only crawls sites upon request from the site owner.
User-authorized website monitoring, uptime checks, diagnostics, and compliance scanning for managed sites.
