Bot intelligence record
Toutiao
Usually allowUse the Bytespider identifier to separate ByteDance search indexing or content discovery traffic from normal visitor requests in server logs.
- Operator
- ByteDance
- Family
- ByteDance
- Type
- Search
- Source type
- Verified Directory
- Last checked
- 2026-05-20
User-Agent Pattern
ByteDanceBytespider
User-agent strings are identification signals, not proof of identity. Confirm important allow, block, or rate-limit decisions with logs, DNS or IP evidence, request behavior, or operator documentation when available.
Robots.txt Snippet
Click snippet to copyUser-agent: zhanzhang.toutiao.com Disallow: /Click the snippet to copy it, or highlight the text manually.
Handling Guidance
DependsThis bot is usually safe to allow when the request source is verified and the traffic matches your site policy.
Search indexing, content discovery, rendering, or search-result freshness checks.
Record Details
Structured data- Operator
- ByteDance
- Family
- ByteDance
- Type
- Search
- Purpose
- Indexing
- Identity type
- Verified Bot
- Confidence
- Medium
- Last verified
- 2026-04-01
- Last checked
- 2026-05-20
- Source type
- Verified Directory
- Verification
- Validate the user-agent pattern, operator documentation, and any published IP ranges before trusting.
- Spoofing risk
- User-agent strings for Bytespider can be spoofed. Treat user-agent detection as a classification signal, then verify with published IP ranges, reverse DNS, signatures, operator documentation, or published operator documentation, IP ranges, reverse DNS, signatures, or other verified identity signals before allow-listing.
Notes
Toutiao is listed in the Botcrawl directory as a search crawler from ByteDance. The primary identifier for log review is Bytespider.
Identification
- User-agent pattern:
Bytespider - Family: ByteDance
- Type: Search
- Kind: Crawler
Common use
Search indexing, content discovery, rendering, or search-result freshness checks.
Verification and handling
Validate the user-agent pattern, operator documentation, and any published IP ranges before trusting.
Directory guidance marks the risk level as Safe and the blocking decision as Depends. Do not rely on the user-agent string alone because user-agent strings can be copied or spoofed.
Robots.txt handling: Yes.
Evidence and Source
- Validate the user-agent pattern, operator documentation, and any published IP ranges before trusting.
- Match `Bytespider` as a case-insensitive substring in HTTP user-agent logs. Review bot_aliases for alternate names or product labels. Use bot_http_agent for full user-agent examples when the client sends a longer browser-like string. Do not treat a user-agent match alone as proof of identity for allow-listing.
- Search indexing, content discovery, rendering, or search-result freshness checks.
- User-agent strings for Bytespider can be spoofed. Treat user-agent detection as a classification signal, then verify with published IP ranges, reverse DNS, signatures, operator documentation, or published operator documentation, IP ranges, reverse DNS, signatures, or other verified identity signals before allow-listing.
Monitor This Bot In Edge
Botcrawl EdgeUse Botcrawl Edge to see matching traffic, create allow or block rules, and control this bot across connected sites.