Bot intelligence record
htdig
Review firsthtdig is a search crawler from ht://Dig used for search indexing, URL discovery, page rendering, result freshness; it appears in server logs as `htdig`.
User-Agent Pattern
ht://Dightdig
User-agent strings are identification signals, not proof of identity. Confirm important allow, block, or rate-limit decisions with logs, DNS or IP evidence, request behavior, or operator documentation when available.
Robots.txt Snippet
Click snippet to copyUser-agent: htdig
Disallow: /Click the snippet to copy it, or highlight the text manually.
Handling Guidance
MonitorUse this record as bot intelligence, then verify the request source and behavior before allowing, blocking, or rate limiting.
htdig is used for search indexing, URL discovery, page rendering, result freshness, and search-quality checks.
Record Details
Structured data- Operator
- ht://Dig
- Family
- ht://Dig
- Purpose
- Search Indexing
- Identity type
- Documented
- Confidence
- Low
- Last verified
- 2026-06-22
- Last checked
- 2026-06-22
- Source type
- Observed
- Verification
- Verify htdig by matching `htdig` to ht://Dig evidence, then checking reverse DNS, source-network ownership, signed request data, or published crawler documentation when available.
- Spoofing risk
- htdig has high spoofing risk because the pattern is low-confidence or observation-based; do not trust the user-agent by itself.
Notes
- htdig is a search crawler from ht://Dig used for search indexing, URL discovery, page rendering, result freshness, and search-quality checks.
- Its primary user-agent pattern is
htdig. - htdig is verified with Low confidence. The identity type is Documented, and the evidence basis is observed traffic patterns and user-agent evidence.
- htdig does not have confirmed robots.txt behavior in the available public evidence.
- htdig should be monitored first, then rate-limited or blocked if the crawl rate, paths, or behavior are unwanted.
Evidence and Source
- Verify htdig by matching `htdig` to ht://Dig evidence, then checking reverse DNS, source-network ownership, signed request data, or published crawler documentation when available.
- htdig traffic is primarily detected by the `htdig` user-agent pattern. Compare source IPs, reverse DNS, request paths, and crawl cadence with ht://Dig infrastructure before trusting the traffic.
- htdig is used for search indexing, URL discovery, page rendering, result freshness, and search-quality checks.
- htdig has high spoofing risk because the pattern is low-confidence or observation-based; do not trust the user-agent by itself.
Monitor This Bot In Edge
Botcrawl EdgeUse Botcrawl Edge to see matching traffic, create allow or block rules, and control this bot across connected sites.