Overview
ICC Crawler is a search crawler from NICT used for search indexing, URL discovery, page rendering, result freshness, and search-quality checks.
Its primary user-agent pattern is ICC-Crawler; related patterns include ICC-Crawler/; a representative HTTP user-agent is ICC-Crawler/3.0 (Mozilla-compatible;; https://ucri.nict.go.jp/en/icccrawler.html).
ICC Crawler is verified with Medium confidence. The identity type is Verified Bot, and the evidence basis is a public crawler reference or source-linked documentation.
ICC Crawler is marked as respecting robots.txt directives for crawler access control.
ICC Crawler should be reviewed against site policy, source evidence, crawl rate, and requested paths before a permanent allow or block rule is created.
Identity
- User-Agent
ICC-Crawler- Aliases
- ICC-Crawler/
- HTTP Agent Examples
ICC-Crawler/3.0 (Mozilla-compatible;; https://ucri.nict.go.jp/en/icccrawler.html)- Robots.txt Token
ICC-Crawler- Identity Type
- Verified Bot
- Evidence Method
- Verify ICC Crawler by matching `ICC-Crawler` to NICT evidence, then checking reverse DNS, source-network ownership, signed request data, or published crawler documentation when available.
Classification
- Type
- Search
- Kind
- Crawler
- Family
- NICT
- Purpose
- indexing
Behavior and handling
- Common Use
- ICC Crawler is used for search indexing, URL discovery, page rendering, result freshness, and search-quality checks.
- Detection Notes
- ICC Crawler traffic is primarily detected by the `ICC-Crawler` user-agent pattern; related patterns include `ICC-Crawler/`; a representative HTTP user-agent is `ICC-Crawler/3.0 (Mozilla-compatible;; https://ucri.nict.go.jp/en/icccrawler.html)`. Compare source IPs, reverse DNS, request paths, and crawl cadence with NICT infrastructure before trusting the traffic.
- Respects robots.txt
- Yes
- Spoofing Risk
- ICC Crawler has medium spoofing risk because user-agent strings can be copied; pair the match with DNS, IP, behavior, or operator evidence.
- Risk
- Neutral
- Recommended Handling
- Depends
Rules and controls
- Robots.txt Snippet
User-agent: ICC-Crawler Disallow: /