Bot intelligence record

Toutiao

Usually allow

Toutiao is a search crawler from ByteDance used for search indexing, URL discovery, page rendering, result freshness; it appears in server logs as `Bytespider`.

Search Indexing Verified Bot Confidence: Medium Verified: Yes robots.txt: Yes
Operator
ByteDance
Family
ByteDance
Type
Search
Source type
Documented
Last checked
2026-06-23

User-Agent Pattern

ByteDance
Bytespider
Verification note

User-agent strings are identification signals, not proof of identity. Confirm important allow, block, or rate-limit decisions with logs, DNS or IP evidence, request behavior, or operator documentation when available.

Robots.txt Snippet

Click snippet to copy 
User-agent: Bytespider
Disallow: /

Click the snippet to copy it, or highlight the text manually.

Handling Guidance

Depends

This bot is usually safe to allow when the request source is verified and the traffic matches your site policy.

Toutiao is used for search indexing, URL discovery, page rendering, result freshness, and search-quality checks.

Record Details

Structured data
Operator
ByteDance
Family
ByteDance
Type
Search
Purpose
Indexing
Identity type
Verified Bot
Confidence
Medium
Last verified
2026-06-23
Last checked
2026-06-23
Source type
Documented
Verification
Verify Toutiao by matching `Bytespider` to ByteDance evidence, then checking reverse DNS, source-network ownership, signed request data, or published crawler documentation when available.
Spoofing risk
Toutiao has medium spoofing risk because user-agent strings can be copied; pair the match with DNS, IP, behavior, or operator evidence.

Notes

  • Toutiao is a search crawler from ByteDance used for search indexing, URL discovery, page rendering, result freshness, and search-quality checks.
  • Its primary user-agent pattern is Bytespider; related patterns include Toutiao crawler, ByteDance crawler; zhanzhang.toutiao.com; Toutiao crawler; ByteDance crawler; a representative HTTP user-agent is Mozilla/5.0 (compatible; Bytespider; https://zhanzhang.toutiao.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.0.0 Safari/537.36.
  • Toutiao is verified with Medium confidence. The identity type is Verified Bot, and the evidence basis is a public crawler reference or source-linked documentation.
  • Toutiao is marked as respecting robots.txt directives for crawler access control.
  • Toutiao can usually be allowed after confirming the source and monitoring request volume.

Evidence and Source

  • Verify Toutiao by matching `Bytespider` to ByteDance evidence, then checking reverse DNS, source-network ownership, signed request data, or published crawler documentation when available.
  • Toutiao traffic is primarily detected by the `Bytespider` user-agent pattern; related patterns include `Toutiao crawler, ByteDance crawler; zhanzhang.toutiao.com; Toutiao crawler; ByteDance crawler`; a representative HTTP user-agent is `Mozilla/5.0 (compatible; Bytespider; https://zhanzhang.toutiao.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.0.0 Safari/537.36`. Compare source IPs, reverse DNS, request paths, and crawl cadence with ByteDance infrastructure before trusting the traffic.
  • Toutiao is used for search indexing, URL discovery, page rendering, result freshness, and search-quality checks.
  • Toutiao has medium spoofing risk because user-agent strings can be copied; pair the match with DNS, IP, behavior, or operator evidence.

https://zhanzhang.toutiao.com/

Same family: ByteDance

TikTok Safe

TikTok

TikTok is a link preview bot from ByteDance used for link unfurling, rich-card generation, social previews, messaging previews.

Operator and family: ByteDance Type: Preview
Verified robots.txt: Yes Block: Depends Verified Bot
View Record Open in Edge BotDB
TikTokSpider Safe

TikTokSpider

TikTokSpider is a web scraper from ByteDance used for public web data collection, page extraction, content monitoring.

Operator and family: ByteDance Type: Scraper Purpose: Scraping
Verified robots.txt: Yes Block: Depends Verified Bot
View Record Open in Edge BotDB

Similar purpose: Indexing

Sidetrade indexer bot Neutral

Sidetrade indexer bot

Sidetrade indexer bot is an AI crawler from Sidetrade used for AI crawler discovery, assistant retrieval, AI search support, or model-related web access.

Operator and family: Sidetrade Type: Ai Purpose: Indexing
Not verified robots.txt: Unknown Block: Depends Observed
View Record Open in Edge BotDB
Qwantbot Safe

Qwantbot

Qwantbot is a search crawler from Qwant used for search indexing, URL discovery, page rendering, result freshness.

Operator and family: Qwant Type: Search Purpose: Indexing
Verified robots.txt: Yes Block: Depends Verified Bot
View Record Open in Edge BotDB
YouBot Safe

YouBot

YouBot is a search crawler from You.com used for search indexing, URL discovery, page rendering, result freshness.

Operator: You.com Family: YouBot Type: Search Purpose: Indexing
Verified robots.txt: No Block: Depends Verified Bot
View Record Open in Edge BotDB
ZoominfoBot Safe

ZoomInfo

ZoomInfo is a search crawler used for search indexing, URL discovery, page rendering, result freshness.

Operator and family: ZoomInfo Type: Search Purpose: Indexing
Verified robots.txt: Unknown Block: Depends Verified Bot
View Record Open in Edge BotDB

Monitor This Bot In Edge

Botcrawl Edge

Use Botcrawl Edge to see matching traffic, identify related datacenter activity, and create allow, block, rate-limit, or log rules across connected sites.

Get Edge Now Open in Edge BotDB