Overview
Google-CWS is a search crawler from Google used for search indexing, URL discovery, page rendering, result freshness, and search-quality checks.
Its primary user-agent pattern is Google-CWS; related patterns include Chrome Web Store fetcher; a representative HTTP user-agent is Mozilla/5.0 (compatible; Google-CWS).
Google-CWS is verified with High confidence. The identity type is Official Documented, and the evidence basis is official operator documentation.
Google-CWS has context-dependent robots.txt behavior because some requests may be user-triggered, product-triggered, or handled differently by the operator.
Google-CWS should be reviewed against site policy, source evidence, crawl rate, and requested paths before a permanent allow or block rule is created.
Identity
- User-Agent
Google-CWS- Aliases
- Chrome Web Store fetcher
- HTTP Agent Examples
Mozilla/5.0 (compatible; Google-CWS)- Robots.txt Token
Google-CWS- Identity Type
- Official Documented
- Evidence Method
- Verify Google-CWS by matching `Google-CWS` to Google evidence, then checking reverse DNS, source-network ownership, signed request data, or published crawler documentation when available.
Classification
- Type
- Fetcher
- Kind
- Fetcher
- Family
- Purpose
- extension-metadata-url-fetching
Behavior and handling
- Common Use
- Fetching URLs supplied by Chrome Web Store developers in extension and theme metadata.
- Detection Notes
- Google-CWS traffic is primarily detected by the `Google-CWS` user-agent pattern; related patterns include `Chrome Web Store fetcher`; a representative HTTP user-agent is `Mozilla/5.0 (compatible; Google-CWS)`. Compare source IPs, reverse DNS, request paths, and crawl cadence with Google infrastructure before trusting the traffic.
- Respects robots.txt
- Depends
- Spoofing Risk
- Google-CWS has medium spoofing risk because the user-agent can be copied, even when the bot has strong source or documentation support.
- Risk
- Neutral
- Recommended Handling
- Depends
Rules and controls
- Robots.txt Snippet
# This rule may not apply to user-triggered or special-purpose requests. User-agent: Google-CWS Disallow: /