webzio-extended
Webz.io extended web crawler that maintains a repository of web crawl data.
robots.txt: Unknown Risky Block: Depends
Webz.io open web crawler that collects data from news, blogs, forums, and social media.
User-agent: webzio
Disallow: /(http.user_agent contains "webzio")RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} webzio [NC]
RewriteRule ^ - [F,L]if ($http_user_agent ~* "webzio") { return 403; }Webz.io open web crawler that collects data from news, blogs, forums, and social media.
webzio is operated by Webz.io. Botcrawl classifies it as a search bot with a primary purpose of site owner fetch.
Key profile signals: risk level: risky; verified: no.
Verify the exact user-agent against Webz.io’s published crawler documentation.
Source documentation: https://webz.io/bot.html
Webz.io extended web crawler that maintains a repository of web crawl data.
Webz.io web crawler that maintains a repository of web crawl data sold to other companies.
WordPress automated testing platform
OKX Dolphin Crawler simulates visits to target web pages as part of malicious dApp scanning
Web classifier: fetches a site’s homepage to assign a category and rechecks periodically.
Robin automatically scans school websites in the UK, providing compliance checks against statutory requirements.