Overview
Meta-WebIndexer is a search crawler from Meta used for search indexing, URL discovery, page rendering, result freshness, and search-quality checks.
Its primary user-agent pattern is meta-webindexer/1.1; a representative HTTP user-agent is meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler) meta-webindexer/1.1.
Meta-WebIndexer is verified with High confidence. The identity type is Verified Bot, and the evidence basis is official operator documentation.
Meta-WebIndexer is marked as respecting robots.txt directives for crawler access control.
Meta-WebIndexer should be reviewed against site policy, source evidence, crawl rate, and requested paths before a permanent allow or block rule is created.
Identity
- User-Agent Pattern
-
meta-webindexer/1.1 - Aliases
- meta-webindexer/1.1
- HTTP Agent Examples
-
meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler) meta-webindexer/1.1 - Robots Token
- meta-webindexer/1.1
- Identity Type
- Verified bot
- Evidence Method
- Verify Meta-WebIndexer by matching `meta-webindexer/1.1` to Meta evidence, then checking reverse DNS, source-network ownership, signed request data, or published crawler documentation when available.
Classification
- Type
- Search
- Kind
- Crawler
- Family
- Meta
- Operator
- Meta
- Company
- Meta
- Purpose
- Indexing
Behavior and handling
- Common Use
- Meta-WebIndexer is used for search indexing, URL discovery, page rendering, result freshness, and search-quality checks.
- Detection Notes
- Meta-WebIndexer traffic is primarily detected by the `meta-webindexer/1.1` user-agent pattern; a representative HTTP user-agent is `meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler) meta-webindexer/1.1`. Compare source IPs, reverse DNS, request paths, and crawl cadence with Meta infrastructure before trusting the traffic.
- Respects robots.txt
- Yes
- Spoofing Risk
- Meta-WebIndexer has medium spoofing risk because the user-agent can be copied, even when the bot has strong source or documentation support.
- Risk
- Neutral
- Recommended Handling
- Depends
Rules and controls
- Robots.txt Snippet
-
User-agent: meta-webindexer/1.1 Disallow: /