Meta-WebIndexer

Review

Meta-WebIndexer is a search crawler from Meta used for search indexing, URL discovery, page rendering, result freshness.

User-Agent

meta-webindexer/1.1

User-agent strings are identity signals, not proof of identity. Use Edge to compare the user-agent with IP, datacenter, behavior, path, and request history.

Handling Guidance

Use this public record as identity context, then verify source, behavior, and site impact inside Edge before creating rules.

Meta-WebIndexer is used for search indexing, URL discovery, page rendering, result freshness, and search-quality checks.

Record Details

Type: search
Operator: Meta
Family: Meta
Kind: Crawler
Purpose: Indexing
Identity: Verified Bot
Detection confidence: High
Respects robots.txt: Yes
Last verified: 2026 06 23

Robots.txt Snippet

User-agent: meta-webindexer/1.1
Disallow: /

Evidence and Source

Source reference
Operator reference