Overview
Siteimprove Crawl is a search crawler from Siteimprove used for search indexing, URL discovery, page rendering, result freshness, and search-quality checks.
Its primary user-agent pattern is SiteCheck-sitecrawl; a representative HTTP user-agent is Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) SiteCheck-sitecrawl.
Siteimprove Crawl is Verified from current authoritative identity documentation reviewed on 2026-08-07. The identity type is Official Documented.
Robots.txt behavior is not currently confirmed.
Siteimprove Crawl should be reviewed against site policy, source evidence, crawl rate, and requested paths before a permanent allow or block rule is created.
Identity
- User-Agent
SiteCheck-sitecrawl- HTTP Agent Examples
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) SiteCheck-sitecrawl- Robots.txt Token
SiteimproveBot-Crawler- Identity Type
- Official Documented
- Evidence Method
- Match `SiteCheck-sitecrawl` to the current operator documentation and corroborate the request with operator-controlled verification signals where available; User-Agent strings alone can be spoofed.
Classification
- Type
- Search
- Kind
- Crawler
- Family
- Siteimprove
- Purpose
- indexing
Behavior and handling
- Common Use
- Siteimprove Crawl is used for search indexing, URL discovery, page rendering, result freshness, and search-quality checks.
- Detection Notes
- Siteimprove Crawl traffic is primarily detected by the `SiteCheck-sitecrawl` user-agent pattern; a representative HTTP user-agent is `Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) SiteCheck-sitecrawl`. Compare source IPs, reverse DNS, request paths, and crawl cadence with Siteimprove infrastructure before trusting the traffic.
- Respects robots.txt
- Unknown
- Spoofing Risk
- Siteimprove Crawl has medium spoofing risk because user-agent strings can be copied; pair the match with DNS, IP, behavior, or operator evidence.
- Risk
- Neutral
- Recommended Handling
- Depends
Rules and controls
- Robots.txt Snippet
# robots.txt behavior is unconfirmed. Do not rely on this rule without verification.