Overview
Biblioteca Nacional de España Web Archive Crawler is an archiver associated with Biblioteca Nacional de España, used for preserving Spanish websites for the national web archive.
Its primary user-agent pattern is bne.es_bot; related patterns include BNE crawler; Spanish Web Archive crawler; a representative HTTP user-agent is Mozilla/5.0 (compatible; bne.es_bot; +https://www.bne.es/es/colecciones/archivo-web-espanola/aviso-webmasters).
Biblioteca Nacional de España Web Archive Crawler is Verified from current authoritative identity documentation reviewed on 2026-08-07. The identity type is Official Documented.
Biblioteca Nacional de España Web Archive Crawler is not governed reliably by robots.txt. Use authenticated access controls, network policy, rate limiting, or application-layer rules when enforcement is required.
The BNE publishes the user-agent template and contact guidance for its Spanish Web Archive crawler. Legal-deposit collection may not follow ordinary robots.txt exclusions.
Treat the user-agent as an identity claim. Corroborate it with operator-controlled IP ranges, reverse DNS, signed-request evidence, or current documentation when available.
Identity
- User-Agent Pattern
-
bne.es_bot - Aliases
- BNE crawler; Spanish Web Archive crawler
- HTTP Agent Examples
-
Mozilla/5.0 (compatible; bne.es_bot; +https://www.bne.es/es/colecciones/archivo-web-espanola/aviso-webmasters) - Robots Token
- bne.es_bot
- Identity Type
- Officially documented
- Evidence Method
- Match `bne.es_bot` to the current operator documentation and corroborate the request with operator-controlled verification signals where available; User-Agent strings alone can be spoofed.
Classification
- Type
- Archive
- Kind
- Archiver
- Family
- Spanish Web Archive
- Purpose
- Legal deposit web archiving
Behavior and handling
- Common Use
- preserving Spanish websites for the national web archive
- Detection Notes
- Detect primarily by `bne.es_bot`; related patterns include `BNE crawler; Spanish Web Archive crawler`; representative HTTP user-agent: `Mozilla/5.0 (compatible; bne.es_bot; +https://www.bne.es/es/colecciones/archivo-web-espanola/aviso-webmasters)`. User-agent strings can be copied, so corroborate with stronger evidence before granting trust.
- Respects robots.txt
- No
- Spoofing Risk
- Biblioteca Nacional de España Web Archive Crawler has medium spoofing risk because its user-agent can be copied; use published operator-controlled verification data when available.
- Risk
- Safe
- Recommended Handling
- Do not block
Rules and controls
- Robots.txt Snippet
-
# This identity is not reliably controlled by robots.txt. Use authenticated access controls, network policy, rate limiting, or application-layer enforcement. - Apache Rule
-
RewriteCond %{HTTP_USER_AGENT} "bne\.es_bot" [NC] RewriteRule ^ - [F,L] - Nginx Rule
-
if ($http_user_agent ~* "bne\.es_bot") { return 403; }
Relationships
- Operator
- Biblioteca Nacional de España Checked 2026-08-07
Relationships without an Evidence link are normalized from the canonical directory record. They should not be interpreted as independent proof of physical presence or request origin.