Overview
FactSet_spyderbot is a search crawler from FactSet used for search indexing, URL discovery, page rendering, result freshness, and search-quality checks.
Its primary user-agent pattern is factset_spyderbot; a representative HTTP user-agent is Mozilla/5.0(windows NT 10.0; Win64; x64) AppleWebkit/537.36(KHTML, like Gecko) Chrome/90.0.4430.72 Safari/537.36 factset_spyderbot.
Factset_spyderbot is Unverified at the identity-evidence level. The listed identity remains useful for detection, but this record does not currently contain authoritative evidence sufficient to authenticate the identity claim.
Robots.txt behavior is not currently confirmed.
FactSet_spyderbot should be reviewed against site policy, source evidence, crawl rate, and requested paths before a permanent allow or block rule is created.
Identity
- User-Agent
factset_spyderbot- HTTP Agent Examples
Mozilla/5.0(windows NT 10.0; Win64; x64) AppleWebkit/537.36(KHTML, like Gecko) Chrome/90.0.4430.72 Safari/537.36 factset_spyderbot- Robots.txt Token
factset_spyderbot- Identity Type
- Observed
- Evidence Method
- Treat `factset_spyderbot` as an identity signal only. Confirm it with current operator documentation, cryptographic verification, forward-confirmed reverse DNS, source-network ownership, or other authoritative evidence before trusting the claimed identity.
Classification
- Type
- Search
- Kind
- Crawler
- Family
- FactSet
- Purpose
- indexing
Behavior and handling
- Common Use
- FactSet_spyderbot is used for search indexing, URL discovery, page rendering, result freshness, and search-quality checks.
- Detection Notes
- FactSet_spyderbot traffic is primarily detected by the `factset_spyderbot` user-agent pattern; a representative HTTP user-agent is `Mozilla/5.0(windows NT 10.0; Win64; x64) AppleWebkit/537.36(KHTML, like Gecko) Chrome/90.0.4430.72 Safari/537.36 factset_spyderbot`. Compare source IPs, reverse DNS, request paths, and crawl cadence with FactSet infrastructure before trusting the traffic.
- Respects robots.txt
- Unknown
- Spoofing Risk
- FactSet_spyderbot has medium spoofing risk because user-agent strings can be copied; pair the match with DNS, IP, behavior, or operator evidence.
- Risk
- Neutral
- Recommended Handling
- Depends
Rules and controls
- Robots.txt Snippet
# robots.txt behavior is unconfirmed. Do not rely on this rule without verification.