Applebot-Extended
Apple control token for foundation-model training opt-out.
Apple web crawler used for search features across Spotlight, Siri, Safari, and related Apple experiences.
Applebot data is used to power various features, such as the search technology that is integrated into many user experiences in Appleʼs ecosystem including Spotlight, Siri, and Safari.
User-agent: Applebot
Disallow: /
(http.user_agent contains "Applebot")
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Applebot [NC]
RewriteRule ^ - [F,L]
if ($http_user_agent ~* "Applebot") { return 403; }
Applebot is Apple’s general search crawler. Apple says the data it crawls powers search features and may also be used to help train foundation models unless you disallow Applebot-Extended.
Apple documents reverse-DNS verification and a JSON list of Applebot IP CIDRs.
Apple control token for foundation-model training opt-out.
The Apple App Site Association is used to support "Universal Links" that can open in native iOS apps.
Apple Podcasts crawler for registered podcast content.
FishBot crawls webpages to deliver Open Source AI for All
The Yext Crawler provides Yext customers with a tool to retrieve data from their own websites.
A content based scraper only for partners we collaborate with who have given permission to have their website scraped.
Yandex crawler for mobile-layout checks.
