About this bot
Meta-WebIndexer is a crawler associated with Meta and used for indexing.
The primary identifying User-Agent token is meta-webindexer/1.1. Representative HTTP request identity: meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler) meta-webindexer/1.1.
Directory status: Active. Identity classification: Verified with High confidence. Identity type: verified bot.
This identity is recorded as honoring robots.txt under meta-webindexer/1.1. Site-specific access policy should still be evaluated before enforcement.
Risk classification: Neutral. Recommended handling is to evaluate the request context before allowing, rate-limiting, or blocking.
How to identify this bot
- User-agent and crawler identifiers
-
meta-webindexermeta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler) - Also known as
- meta-webindexer/1.1
Details
- Kind
- Search crawler
- Family
- Meta
- Purpose
- Meta AI search indexing
- Common Use
- Building and refreshing the web index used by Meta AI search and retrieval experiences.
- Respects robots.txt
- Yes
Verification
- Identity Type
- Verified bot
- Evidence Method
- Match `meta-webindexer/1.1` as the claimed identity signal and corroborate it with independent network, DNS, cryptographic, or request-context signals when stronger authentication is required.
- Detection Notes
- Match the `meta-webindexer/1.1` User-Agent token and consider recorded aliases where applicable. A User-Agent match identifies a claim, not authenticated origin.
Website controls
- Robots.txt Snippet
User-agent: meta-webindexer Disallow: /