About this bot
kb.dk_bot is a crawler-service associated with Royal Danish Library and used for national-web-archiving.
This record does not rely on one fixed bot User-Agent. Request identity: Mozilla/5.0 (compatible; heritrix/3.10.0 +https://www.kb.dk/netarkivindsamling) Firefox/144.
Directory status: Active. Identity classification: Verified with High confidence.
robots.txt is not a reliable control for this traffic; use server-side controls where restriction is required.
Netarkivet performs Danish legal-deposit web archiving with Heritrix. The service intentionally does not use robots.txt as an access-control policy.
How to identify this bot
- User-agent and crawler identifiers
-
Mozilla/5.0 (compatible; heritrix/3.10.0 +https://www.kb.dk/netarkivindsamling) Firefox/144 - Also known as
- kb.dk_bot; netarkivindsamling kb.dk_bot netarkivindsamling
Details
- Kind
- Crawler service
- Family
- Netarkivet
- Purpose
- National web archiving
- Common Use
- Netarkivet performs Danish legal-deposit web archiving with Heritrix. The service intentionally does not use robots.txt as an access-control policy.
- Respects robots.txt
- No
Verification
- Identity Type
- service-family
- Evidence Method
- Treat `Netarkivet web archive crawler` as a documented service-family identity. Verify the product-specific request pattern rather than assuming one fixed User-Agent.
- Detection Notes
- Detect the service using its documented request patterns and product-specific identities; `Netarkivet web archive crawler` is a service-family label, not necessarily a literal HTTP User-Agent.
Website controls
- Robots.txt Snippet
# robots.txt is not a reliable control for this identity. Use server-side access controls if restriction is required.