Overview
OnCrawl is a search crawler used for search indexing, URL discovery, page rendering, result freshness, and search-quality checks.
Its primary user-agent pattern is OnCrawl Mobile/1.0; related patterns include Oncrawl crawler; www.oncrawl.com; a representative HTTP user-agent is Mozilla/5.0 (compatible; OnCrawl Mobile/1.0; +http://www.oncrawl.com/).
OnCrawl is verified with Medium confidence. The identity type is Verified Bot, and the evidence basis is a public crawler reference or source-linked documentation.
OnCrawl is marked as respecting robots.txt directives for crawler access control.
OnCrawl can usually be allowed after confirming the source and monitoring request volume.
Identity
- User-Agent
OnCrawl Mobile/1.0- Aliases
- Oncrawl crawler; www.oncrawl.com
- HTTP Agent Examples
Mozilla/5.0 (compatible; OnCrawl Mobile/1.0; +http://www.oncrawl.com/)- Robots.txt Token
OnCrawl Mobile/1.0- Identity Type
- Verified Bot
- Evidence Method
- Verify OnCrawl by matching `OnCrawl Mobile/1.0` to a public crawler reference or source-linked documentation, then checking reverse DNS, IP ownership, request behavior, and crawl consistency.
Classification
- Type
- Search
- Kind
- Crawler
- Family
- Oncrawl
- Purpose
- indexing
Behavior and handling
- Common Use
- OnCrawl is used for search indexing, URL discovery, page rendering, result freshness, and search-quality checks.
- Detection Notes
- OnCrawl traffic is primarily detected by the `OnCrawl Mobile/1.0` user-agent pattern; related patterns include `Oncrawl crawler; www.oncrawl.com`; a representative HTTP user-agent is `Mozilla/5.0 (compatible; OnCrawl Mobile/1.0; +http://www.oncrawl.com/)`. Compare source IPs, reverse DNS, request paths, and crawl cadence before trusting the traffic.
- Respects robots.txt
- Yes
- Spoofing Risk
- OnCrawl has medium spoofing risk because user-agent strings can be copied; pair the match with DNS, IP, behavior, or operator evidence.
- Risk
- Safe
- Recommended Handling
- Depends
Rules and controls
- Robots.txt Snippet
User-agent: OnCrawl Mobile/1.0 Disallow: /