webzio
Webz.io open web crawler that collects data from news, blogs, forums, and social media.
robots.txt: Unknown Risky Block: Depends
Webz.io web crawler that maintains a repository of web crawl data sold to other companies.
User-agent: omgili
Disallow: /(http.user_agent contains "omgili")RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} omgili [NC]
RewriteRule ^ - [F,L]if ($http_user_agent ~* "omgili") { return 403; }Webz.io web crawler that maintains a repository of web crawl data sold to other companies.
omgili is operated by Webz.io. Botcrawl classifies it as a ai bot with a primary purpose of site owner fetch.
Key profile signals: risk level: risky; verified: no.
Verify the exact user-agent against Webz.io’s published crawler documentation.
Source documentation: https://webz.io
Webz.io open web crawler that collects data from news, blogs, forums, and social media.
Webz.io extended web crawler that maintains a repository of web crawl data.
WordPress automated testing platform
OKX Dolphin Crawler simulates visits to target web pages as part of malicious dApp scanning
Web classifier: fetches a site’s homepage to assign a category and rechecks periodically.
Robin automatically scans school websites in the UK, providing compliance checks against statutory requirements.