About this bot
Library Of Congress Web Archiving is a crawler-service associated with United States Library of Congress and used for web-archiving.
This record does not rely on one fixed bot User-Agent. Request identity: Variable crawler identity; the Library of Congress advises site owners that crawler User-Agent and network details can change..
Directory status: Active. Identity classification: Verified with High confidence.
robots.txt is not a reliable control for this traffic; use server-side controls where restriction is required.
The Library of Congress uses web crawlers, primarily Heritrix-based workflows, to preserve selected web content and generally bypasses robots.txt for archival capture. A fake fixed URL-as-User-Agent has been removed.
How to identify this bot
- User-agent and crawler identifiers
-
Variable crawler identity; the Library of Congress advises site owners that crawler User-Agent and network details can change. - Also known as
- LOC Web Archiving
Details
- Kind
- Crawler service
- Family
- Library of Congress Web Archiving
- Purpose
- Web archiving
- Common Use
- The Library of Congress uses web crawlers, primarily Heritrix-based workflows, to preserve selected web content and generally bypasses robots.txt for archival capture. A fake fixed URL-as-User-Agent has been removed.
- Respects robots.txt
- No
Verification
- Identity Type
- service-family
- Evidence Method
- Treat `Library of Congress Web Archiving` as a documented service-family identity. Verify the product-specific request pattern rather than assuming one fixed User-Agent.
- Detection Notes
- Detect the service using its documented request patterns and product-specific identities; `Library of Congress Web Archiving` is a service-family label, not necessarily a literal HTTP User-Agent.
Website controls
- Robots.txt Snippet
# robots.txt is not a reliable control for this identity. Use server-side access controls if restriction is required.