About this bot
Deepcrawl is associated with Lumar and used for technical seo and website intelligence.
The primary identifying User-Agent token is lumar. Representative HTTP request identity: Browser-compatible User-Agent containing `lumar` or the historical `deepcrawl` token.
Directory status: Active. Identity classification: Verified with High confidence.
robots.txt behavior: yes under lumar.
Crawls customer-selected websites for technical SEO auditing, site intelligence, accessibility and website health analysis.
How to identify this bot
- User-agent and crawler identifiers
-
lumarBrowser-compatible User-Agent containing `lumar` or the historical `deepcrawl` token - Also known as
- deepcrawl; Deepcrawl; Lumar deepcrawl
Details
- Kind
- Site audit crawler
- Family
- Lumar / Deepcrawl
- Purpose
- Technical SEO and website intelligence
- Common Use
- Crawls customer-selected websites for technical SEO auditing, site intelligence, accessibility and website health analysis.
- Respects robots.txt
- Yes
Verification
- Identity Type
- Officially documented
- Evidence Method
- Match Lumar's documented `lumar`/`deepcrawl` crawler token and corroborate with Lumar's published crawler source addresses where stronger authentication is needed.
- Detection Notes
- Lumar is the current product/operator name. Existing customers may still generate traffic carrying the historical `deepcrawl` token, so both are retained.
Website controls
- Robots.txt Snippet
User-agent: lumar Disallow: /- Apache Rule
RewriteCond %{HTTP_USER_AGENT} "lumar" [NC] RewriteRule ^ - [F,L]- Nginx Rule
if ($http_user_agent ~* "lumar") { return 403; }