Overview
Sogou web spider is operated by Sogou and is used for Crawling web pages for Sogou Search.
Its documented identity is Sogou web spider/4.0. The robots.txt token is Sogou web spider.
It follows robots.txt rules addressed to Sogou web spider.
Match the case-sensitive Sogou web spider token and corroborate it with current Sogou documentation, source-network ownership, request paths, and crawl behavior.
Identity
- User-Agent Pattern
-
Sogou web spider - Aliases
- Sogou web spider/4.0; Sogou Spider
- HTTP Agent Examples
-
Sogou web spider/4.0 - Robots Token
- Sogou web spider
- Identity Type
- Officially documented
- Evidence Method
- Match the case-sensitive Sogou web spider token and corroborate it with current Sogou documentation, source-network ownership, request paths, and crawl behavior.
Classification
- Type
- Search
- Kind
- Crawler
- Family
- Sogou
- Operator
- Sogou
- Company
- Sogou Inc.
- Purpose
- Search indexing
Behavior and handling
- Common Use
- Crawling web pages for Sogou Search.
- Detection Notes
- Sogou documents both Sogou web spider and Sogou inst spider and states that its spiders support robots.txt. The user-agent remains spoofable, so behavioral and network corroboration are necessary.
- Respects robots.txt
- Yes
- Spoofing Risk
- The plain-text user agent is easy to copy; network and behavioral evidence are required.
- Risk
- Neutral
- Recommended Handling
- Depends
Rules and controls
- Robots.txt Snippet
-
User-agent: Sogou web spider Disallow: / - Apache Rule
-
RewriteCond %{HTTP_USER_AGENT} Sogou\ web\ spider [NC] RewriteRule .* - [F,L] - Nginx Rule
-
if ($http_user_agent ~* "Sogou\ web\ spider") { return 403; }