Bot intelligence record

Google-CWS

Review first

Use the Google-CWS identifier to separate Google search indexing or content discovery traffic from normal visitor requests in server logs.

Search Official Documented Confidence: High Verified: Yes robots.txt: No
Operator
Google
Family
Google
Type
Search
Source type
Official
Last checked
2026-05-20

User-Agent Pattern

Google
Google-CWS
Verification note

User-agent strings are identification signals, not proof of identity. Confirm important allow, block, or rate-limit decisions with logs, DNS or IP evidence, request behavior, or operator documentation when available.

Robots.txt Snippet

Click snippet to copy 
User-agent: Google-CWS Disallow: /

Click the snippet to copy it, or highlight the text manually.

Handling Guidance

Depends

Use this record as bot intelligence, then verify the request source and behavior before allowing, blocking, or rate limiting.

Search indexing, content discovery, rendering, or search-result freshness checks.

Record Details

Structured data
Operator
Google
Family
Google
Type
Search
Purpose
Search
Identity type
Official Documented
Confidence
High
Last verified
2026-04-01
Last checked
2026-05-20
Source type
Official
Verification
Published Google crawler documentation and Google crawler/fetcher verification guidance
Spoofing risk
User-agent strings for Google-CWS can be spoofed. Treat user-agent detection as a classification signal, then verify with published IP ranges, reverse DNS, signatures, operator documentation, or published operator documentation, IP ranges, reverse DNS, signatures, or other verified identity signals before allow-listing.

Notes

Google-CWS is listed in the Botcrawl directory as a search crawler from Google. The primary identifier for log review is Google-CWS.

Identification

  • User-agent pattern: Google-CWS
  • Family: Google
  • Type: Search
  • Kind: Fetcher

Common use

Search indexing, content discovery, rendering, or search-result freshness checks.

Verification and handling

Published Google crawler documentation and Google crawler/fetcher verification guidance

Directory guidance marks the risk level as Neutral and the blocking decision as Depends. Do not rely on the user-agent string alone because user-agent strings can be copied or spoofed.

Robots.txt handling: No.

Evidence and Source

  • Published Google crawler documentation and Google crawler/fetcher verification guidance
  • Match `Google-CWS` as a case-insensitive substring in HTTP user-agent logs. Review bot_aliases for alternate names or product labels. Use bot_http_agent for full user-agent examples when the client sends a longer browser-like string. Do not treat a user-agent match alone as proof of identity for allow-listing.
  • Search indexing, content discovery, rendering, or search-result freshness checks.
  • User-agent strings for Google-CWS can be spoofed. Treat user-agent detection as a classification signal, then verify with published IP ranges, reverse DNS, signatures, operator documentation, or published operator documentation, IP ranges, reverse DNS, signatures, or other verified identity signals before allow-listing.

https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers

Same family: Google

ProjectShield-UrlCheck Safe

ProjectShield Url Check

ProjectShield Url Check is a security scanner from Google used for security checks, malware review, or abuse prevention.

Operator and family: Google Type: Security
Verified robots.txt: No Block: No Verified Bot
GoogleOther-Image Neutral

GoogleOther-Image

GoogleOther-Image is an AI crawler from Google used for AI-related crawling, retrieval, search, or assistant activity.

Operator and family: Google Type: Ai Purpose: Ai Training
Verified robots.txt: Yes Block: Depends Official Documented
GoogleOther-Video Neutral

GoogleOther-Video

GoogleOther-Video is an AI crawler from Google used for AI-related crawling, retrieval, search, or assistant activity.

Operator and family: Google Type: Ai Purpose: Ai Training
Verified robots.txt: Yes Block: Depends Official Documented
Google-Pinpoint Safe

Google-Pinpoint

Google-Pinpoint is a search crawler from Google used for search indexing, page discovery, or content rendering.

Operator and family: Google Type: Search
Verified robots.txt: No Block: Depends Official Documented

Same operator: Google

GoogleAgent-Mariner Neutral

GoogleAgent-Mariner

Claimed Google Project Mariner user-agent token for agentic browser activity.

Operator: Google Family: Google Mariner Type: Ai Purpose: Agentic Browsing
Not verified robots.txt: Unknown Block: Depends Observed
Google-Display-Ads-Bot Neutral

Google-Display-Ads-Bot

Google AdSense uses a crawler called Google-Display-Ads-Bot to verify your site when you add a site to AdSense.

Operator and family: Google Type: Search Purpose: Indexing
Verified robots.txt: Yes Block: Depends Verified Bot
Google-Extended Safe

Google-Extended

Google-Extended is an AI control token from Google used for AI-related crawling, retrieval, search, or assistant activity.

Operator and family: Google Type: Ai Purpose: Ai Assistant
Verified robots.txt: Yes Block: Depends Official Documented
Google-NotebookLM Neutral

Google-NotebookLM

Google-NotebookLM is an AI fetcher from Google used for AI-related crawling, retrieval, search, or assistant activity.

Operator and family: Google Type: Ai Purpose: Ai Assistant
Verified robots.txt: No Block: Depends Official Documented

Similar purpose: Search

YandexUserproxy Safe

YandexUserproxy

YandexUserproxy is a search crawler from Yandex used for search indexing, page discovery, or content rendering.

Operator and family: Yandex Type: Search
Verified robots.txt: No Block: Depends Official Documented
YandexSearchShop Safe

YandexSearchShop

YandexSearchShop is a search crawler from Yandex used for search indexing, page discovery, or content rendering.

Operator and family: Yandex Type: Search
Verified robots.txt: No Block: Depends Official Documented
YandexCalendar Safe

YandexCalendar

YandexCalendar is a search crawler from Yandex used for search indexing, page discovery, or content rendering.

Operator and family: Yandex Type: Search
Verified robots.txt: No Block: Depends Official Documented
Qwantbot-news Safe

Qwantbot-news

Qwantbot-news is a search crawler from Qwant used for search indexing, page discovery, or content rendering.

Operator and family: Qwant Type: Search
Verified robots.txt: Yes Block: Depends Official Documented

Monitor This Bot In Edge

Botcrawl Edge

Use Botcrawl Edge to see matching traffic, create allow or block rules, and control this bot across connected sites.

Get Edge Now Open BotDB