Bot intelligence record

Google-CWS

Review first

Use the Google-CWS identifier to separate Google search indexing or content discovery traffic from normal visitor requests in server logs.

Search Search Official Documented Confidence: High Verified: Yes robots.txt: No

User-Agent Pattern

Google
Google-CWS

Robots.txt Snippet

Copy-ready 
User-agent: Google-CWS Disallow: /

Handling Guidance

Depends

Use this record as bot intelligence, then verify the request source and behavior before allowing, blocking, or rate limiting.

Search indexing, content discovery, rendering, or search-result freshness checks.

Record Details

Structured data
Operator
Google
Family
Google
Type
Search
Purpose
Search
Identity type
Official Documented
Confidence
High
Last verified
2026-04-01
Last checked
2026-04-30
Source type
Official

Notes

Google-CWS is listed in the Botcrawl directory as a search crawler from Google. The primary identifier for log review is Google-CWS.

Identification

  • User-agent pattern: Google-CWS
  • Family: Google
  • Type: Search
  • Kind: Fetcher

Common use

Search indexing, content discovery, rendering, or search-result freshness checks.

Verification and handling

Published Google crawler documentation and Google crawler/fetcher verification guidance

Directory guidance marks the risk level as Neutral and the blocking decision as Depends. Do not rely on the user-agent string alone because user-agent strings can be copied or spoofed.

Robots.txt handling: No.

Evidence and Source

  • Published Google crawler documentation and Google crawler/fetcher verification guidance
  • Search indexing, content discovery, rendering, or search-result freshness checks.

https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers

Same family: Google

ProjectShield-UrlCheck Safe

ProjectShield Url Check

ProjectShield Url Check is a security scanner from Google used for security checks, malware review, or abuse prevention.

Google Google Security Security
Verified robots.txt: No Block: No Verified Bot
GoogleOther Neutral

GoogleOther

Generic crawler that may be used by various product teams for fetching publicly accessible content from sites. For example, it may be used for one-off crawls for internal research and.

Google Google Search Indexing
Verified robots.txt: Yes Block: Depends Official Documented
GoogleOther-Image Neutral

GoogleOther-Image

GoogleOther-Image is an AI crawler from Google used for AI-related crawling, retrieval, search, or assistant activity.

Google Google Ai Ai Training
Verified robots.txt: Yes Block: Depends Official Documented
GoogleMessages Safe

GoogleMessages

GoogleMessages is a preview or prefetch bot from Google used to fetch pages for previews, unfurls, or user-triggered rendering.

Google Google Preview Preview
Verified robots.txt: No Block: Depends Official Documented

Same operator: Google

GoogleOther-Video Neutral

GoogleOther-Video

GoogleOther-Video is an AI crawler from Google used for AI-related crawling, retrieval, search, or assistant activity.

Google Google Ai Ai Training
Verified robots.txt: Yes Block: Depends Official Documented
Google-Extended Safe

Google-Extended

Google-Extended is an AI control token from Google used for AI-related crawling, retrieval, search, or assistant activity.

Google Google Ai Ai Assistant
Verified robots.txt: Yes Block: Depends Official Documented
Google-NotebookLM Neutral

Google-NotebookLM

Google-NotebookLM is an AI fetcher from Google used for AI-related crawling, retrieval, search, or assistant activity.

Google Google Ai Ai Assistant
Verified robots.txt: No Block: Depends Official Documented
Google-Pinpoint Safe

Google-Pinpoint

Google-Pinpoint is a search crawler from Google used for search indexing, page discovery, or content rendering.

Google Google Search Search
Verified robots.txt: No Block: Depends Official Documented

Similar purpose: Search

YandexUserproxy Safe

YandexUserproxy

YandexUserproxy is a search crawler from Yandex used for search indexing, page discovery, or content rendering.

Yandex Yandex Search Search
Verified robots.txt: No Block: Depends Official Documented
YandexSearchShop Safe

YandexSearchShop

YandexSearchShop is a search crawler from Yandex used for search indexing, page discovery, or content rendering.

Yandex Yandex Search Search
Verified robots.txt: No Block: Depends Official Documented
YandexCalendar Safe

YandexCalendar

YandexCalendar is a search crawler from Yandex used for search indexing, page discovery, or content rendering.

Yandex Yandex Search Search
Verified robots.txt: No Block: Depends Official Documented
Qwantbot-news Safe

Qwantbot-news

Qwantbot-news is a search crawler from Qwant used for search indexing, page discovery, or content rendering.

Qwant Qwant Search Search
Verified robots.txt: Yes Block: Depends Official Documented

WordPress Bot Protection

Bot Blocker for WordPress

Monitor bot traffic, review live activity, and control AI crawlers, scrapers, scanners, spam bots, and fake trusted bots from one clean WordPress dashboard.

Buy Now Start Free Trial