About this bot
NostoCrawlerBot is a crawler associated with Nosto and used for indexing.
The primary identifying User-Agent token is NostoCrawlerBot. Representative HTTP request identity: Mozilla/5.0 (compatible; NostoCrawlerBot/1.0; +http://my.nosto.com/tagging).
Directory status: Active. Identity classification: Verified with Medium confidence. Identity type: verified bot.
robots.txt is not recorded as a reliable control for this identity. Use server-side controls when restriction is required.
Risk classification: Neutral. Recommended handling is to evaluate the request context before allowing, rate-limiting, or blocking.
How to identify this bot
- User-agent and crawler identifiers
-
NostoCrawlerBotMozilla/5.0 (compatible; NostoCrawlerBot/1.0; +http://my.nosto.com/tagging)
Details
- Kind
- Crawler
- Family
- Nosto
- Purpose
- Indexing
- Common Use
- NostoCrawlerBot is used for search indexing, URL discovery, page rendering, result freshness, and search-quality checks.
- Respects robots.txt
- No
Verification
- Identity Type
- Verified bot
- Evidence Method
- Match `NostoCrawlerBot` as the claimed identity signal and corroborate it with independent network, DNS, cryptographic, or request-context signals when stronger authentication is required.
- Detection Notes
- Match the `NostoCrawlerBot` User-Agent token. A User-Agent match identifies a claim, not authenticated origin.
Website controls
- Robots.txt Snippet
# robots.txt is not a reliable control for this identity. Use server-side access controls if restriction is required.