Overview
AIWebIndex is a crawler associated with Lyrenth, used for building an AI-readable index of public web pages.
Its primary user-agent pattern is AIWebIndex; related patterns include AIWebIndex/2.0; a representative HTTP user-agent is AIWebIndex/2.0 (+https://lyrenth.com/bot; AI-readable web index).
AIWebIndex is verified with High confidence. The identity type is Verified Bot.
AIWebIndex is documented or strongly corroborated as honoring robots.txt directives.
The operator publishes user-agent formats, robots.txt behavior, IP ranges, reverse-DNS guidance, and Web Bot Auth or ownership-verification mechanisms.
Treat the user-agent as an identity claim. Corroborate it with operator-controlled IP ranges, reverse DNS, signed-request evidence, or current documentation when available.
Identity
- User-Agent Pattern
-
AIWebIndex - Aliases
- AIWebIndex/2.0
- HTTP Agent Examples
-
AIWebIndex/2.0 (+https://lyrenth.com/bot; AI-readable web index) - Robots Token
- AIWebIndex
- Identity Type
- Verified bot
- Evidence Method
- Match `AIWebIndex` and validate the source using Lyrenth's published IP ranges, forward-confirmed reverse DNS, Web Bot Auth, or ownership-verification documentation.
Classification
- Type
- AI
- Kind
- Crawler
- Family
- AIWebIndex
- Purpose
- AI search indexing
Behavior and handling
- Common Use
- building an AI-readable index of public web pages
- Detection Notes
- Detect primarily by `AIWebIndex`; related patterns include `AIWebIndex/2.0`; representative HTTP user-agent: `AIWebIndex/2.0 (+https://lyrenth.com/bot; AI-readable web index)`. User-agent strings can be copied, so corroborate with stronger evidence before granting trust.
- Respects robots.txt
- Yes
- Spoofing Risk
- AIWebIndex has medium spoofing risk because its user-agent can be copied; use published operator-controlled verification data when available.
- Risk
- Neutral
- Recommended Handling
- Depends
Rules and controls
- Robots.txt Snippet
-
User-agent: AIWebIndex Disallow: / - Apache Rule
-
RewriteCond %{HTTP_USER_AGENT} "AIWebIndex" [NC] RewriteRule ^ - [F,L] - Nginx Rule
-
if ($http_user_agent ~* "AIWebIndex") { return 403; }
Relationships
- Operator
- Lyrenth Checked 2026-08-02
Relationships without an Evidence link are normalized from the canonical directory record. They should not be interpreted as independent proof of physical presence or request origin.