Overview
SemrushBotSI is a search crawler from Semrush used for search indexing, URL discovery, page rendering, result freshness, and search-quality checks.
Its primary user-agent pattern is SemrushBot-SI; related patterns include Semrush on-page SEO crawler; a representative HTTP user-agent is Mozilla/5.0 (compatible; SemrushBot-SI/0.97; +http://www.semrush.com/bot.html).
SemrushBotSI is verified with High confidence. The identity type is Verified Bot, and the evidence basis is official operator documentation.
SemrushBotSI is marked as respecting robots.txt directives for crawler access control.
SemrushBotSI should be reviewed against site policy, source evidence, crawl rate, and requested paths before a permanent allow or block rule is created.
Identity
- User-Agent
SemrushBot-SI- Aliases
- Semrush on-page SEO crawler
- HTTP Agent Examples
Mozilla/5.0 (compatible; SemrushBot-SI/0.97; +http://www.semrush.com/bot.html)- Robots.txt Token
SemrushBot-SI- Identity Type
- Verified Bot
- Evidence Method
- Verify SemrushBotSI by matching `SemrushBot-SI` to Semrush evidence, then checking reverse DNS, source-network ownership, signed request data, or published crawler documentation when available.
Classification
- Type
- Search
- Kind
- Crawler
- Family
- Semrush
- Purpose
- indexing
Behavior and handling
- Common Use
- SemrushBotSI is used for search indexing, URL discovery, page rendering, result freshness, and search-quality checks.
- Detection Notes
- SemrushBotSI traffic is primarily detected by the `SemrushBot-SI` user-agent pattern; related patterns include `Semrush on-page SEO crawler`; a representative HTTP user-agent is `Mozilla/5.0 (compatible; SemrushBot-SI/0.97; +http://www.semrush.com/bot.html)`. Compare source IPs, reverse DNS, request paths, and crawl cadence with Semrush infrastructure before trusting the traffic.
- Respects robots.txt
- Yes
- Spoofing Risk
- SemrushBotSI has medium spoofing risk because the user-agent can be copied, even when the bot has strong source or documentation support.
- Risk
- Neutral
- Recommended Handling
- Depends
Rules and controls
- Robots.txt Snippet
User-agent: SemrushBot-SI Disallow: /