webzio
Webz.io open web crawler that collects data from news, blogs, forums, and social media.
Webz.io extended web crawler that maintains a repository of web crawl data.
User-agent: webzio-extended
Disallow: /
(http.user_agent contains "webzio-extended")
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} webzio\-extended [NC]
RewriteRule ^ - [F,L]
if ($http_user_agent ~* "webzio-extended") { return 403; }
Webz.io extended web crawler that maintains a repository of web crawl data.
webzio-extended is operated by Webz.io. Botcrawl classifies it as a ai bot with a primary purpose of control token.
Key profile signals: risk level: risky; verified: no.
Verify the exact user-agent against Webz.io’s published crawler documentation.
Source documentation: https://webz.io/bot.html
Webz.io open web crawler that collects data from news, blogs, forums, and social media.
Webz.io web crawler that maintains a repository of web crawl data sold to other companies.
Google control token for Gemini training and grounding permissions.
Apple control token for foundation-model training opt-out.
