About this bot
Toutiao is an automated identity associated with ByteDance and used for discovering, fetching, and indexing webpages for Toutiao Search.
The primary recorded identity signal is Bytespider. Representative HTTP identity: Mozilla/5.0 (compatible; Bytespider; https://zhanzhang.toutiao.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.0.0 Safari/537.36.
Directory status: Active. Identity classification: Verified with High confidence. Identity type: network-verifiable-crawler.
robots.txt behavior is not currently confirmed; use server-side controls when enforcement is required.
Risk classification: Neutral. Recommended handling: depends.
How to identify this bot
- User-agent and crawler identifiers
-
BytespiderMozilla/5.0 (compatible; Bytespider; https://zhanzhang.toutiao.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.0.0 Safari/537.36 - Also known as
- Toutiao crawler; ByteDance crawler
Details
- Kind
- Search crawler
- Family
- ByteDance
- Purpose
- Toutiao search indexing
- Common Use
- discovering, fetching, and indexing webpages for Toutiao Search
- Respects robots.txt
- Unknown
Verification
- Identity Type
- network-verifiable-crawler
- Evidence Method
- Match the Bytespider token and verify the request with ByteDance crawler network/rDNS signals, including a genuine hostname under the documented ByteDance crawler domain that resolves back to the source IP.
- Detection Notes
- ByteDance’s webmaster documentation describes Bytespider as a Toutiao Search crawler and documents source-network/reverse-DNS verification. Robots compliance is left unknown here rather than overclaimed.
Website controls
- Robots.txt Snippet
# robots.txt behavior is unconfirmed. Do not rely on a robots.txt rule without verification.- Apache Rule
SetEnvIfNoCase User-Agent "Bytespider" bot_match- Nginx Rule
if ($http_user_agent ~* "Bytespider") { set $bot_match 1; }