ProjectShield Url Check
Project Shield, created by Google Cloud and Jigsaw and powered by Google Cloud Armor, provides free unlimited protection against DDoS attacks, a type of digital attack…
Chrome-Lighthouse is an automated, open-source tool for auditing web page quality and does not operate as a traditional web crawler. It runs a series of audits against a given page to generate a report on performance and accessibility.
User-agent: Chrome-Lighthouse
Disallow: /
(http.user_agent contains "Chrome-Lighthouse")
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Chrome\-Lighthouse [NC]
RewriteRule ^ - [F,L]
if ($http_user_agent ~* "Chrome-Lighthouse") { return 403; }
It runs a series of audits against a given page to generate a report on performance and accessibility.
Known user-agent patterns: Chrome-Lighthouse Known user-agent strings: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5 Build/MRA58N) AppleWebKit/537.36(KHTML, like Gecko) Chrome/69.0.3464.0 Mobile Safari/537.36 Chrome-Lighthouse Robots.txt handling in the directory: no.
Operator documentation: https://developers.google.com/web/tools/lighthouse
Project Shield, created by Google Cloud and Jigsaw and powered by Google Cloud Armor, provides free unlimited protection against DDoS attacks, a type of digital attack…
Google Messages link-preview fetcher.
Generic Google crawler for public-content fetches outside core Search.
GoogleOther-Image is GoogleOther optimized for fetching publicly accessible image URLs.
GoogleOther-Video is GoogleOther optimized for fetching publicly accessible video URLs.
Google crawler used on owner request for Vertex AI Agents.
Google AdSense uses a crawler called Google-Display-Ads-Bot to verify your site when you add a site to AdSense.
Google-CWS is the Chrome Web Store user-triggered fetcher for URLs developers provide in extension and theme metadata.
WordPress automated testing platform
OKX Dolphin Crawler simulates visits to target web pages as part of malicious dApp scanning
Web classifier: fetches a site’s homepage to assign a category and rechecks periodically.
Robin automatically scans school websites in the UK, providing compliance checks against statutory requirements.
