Overview
Toutiao is a search crawler from ByteDance used for search indexing, URL discovery, page rendering, result freshness, and search-quality checks.
Its primary user-agent pattern is Bytespider; related patterns include Toutiao crawler, ByteDance crawler; zhanzhang.toutiao.com; Toutiao crawler; ByteDance crawler; a representative HTTP user-agent is Mozilla/5.0 (compatible; Bytespider; https://zhanzhang.toutiao.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.0.0 Safari/537.36.
Toutiao is Unverified at the identity-evidence level. The listed identity remains useful for detection, but this record does not currently contain authoritative evidence sufficient to authenticate the identity claim.
Toutiao is marked as respecting robots.txt directives for crawler access control.
Toutiao can usually be allowed after confirming the source and monitoring request volume.
Identity
- User-Agent Pattern
-
Bytespider - Aliases
- Toutiao crawler, ByteDance crawler; zhanzhang.toutiao.com; Toutiao crawler; ByteDance crawler
- HTTP Agent Examples
-
Mozilla/5.0 (compatible; Bytespider; https://zhanzhang.toutiao.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.0.0 Safari/537.36 - Robots Token
- Bytespider
- Identity Type
- Observed
- Evidence Method
- Treat `Bytespider` as an identity signal only. Confirm it with current operator documentation, cryptographic verification, forward-confirmed reverse DNS, source-network ownership, or other authoritative evidence before trusting the claimed identity.
Classification
- Type
- Search
- Kind
- Crawler
- Family
- ByteDance
- Operator
- ByteDance
- Company
- ByteDance
- Purpose
- Indexing
Behavior and handling
- Common Use
- Toutiao is used for search indexing, URL discovery, page rendering, result freshness, and search-quality checks.
- Detection Notes
- Toutiao traffic is primarily detected by the `Bytespider` user-agent pattern; related patterns include `Toutiao crawler, ByteDance crawler; zhanzhang.toutiao.com; Toutiao crawler; ByteDance crawler`; a representative HTTP user-agent is `Mozilla/5.0 (compatible; Bytespider; https://zhanzhang.toutiao.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.0.0 Safari/537.36`. Compare source IPs, reverse DNS, request paths, and crawl cadence with ByteDance infrastructure before trusting the traffic.
- Respects robots.txt
- Yes
- Spoofing Risk
- Toutiao has medium spoofing risk because user-agent strings can be copied; pair the match with DNS, IP, behavior, or operator evidence.
- Risk
- Safe
- Recommended Handling
- Depends
Rules and controls
- Robots.txt Snippet
-
User-agent: Bytespider Disallow: /