Bot Directory

GoogleProducer Safe

Google Publisher Center

Fetches and processes feeds that publishers explicitly supplied for use in Google News landing pages.

Operator and family: Google Type: Feed Purpose: Feed Fetch

Verified robots.txt: No Block: Depends Verified Bot

Google-Read-Aloud Safe

Google Read Aloud

Google Read Aloud service enables reading web pages using text-to-speech (TTS).

Operator and family: Google Type: Monitoring

Verified robots.txt: No Block: No Verified Bot

Google-Safety Safe

Google Safety

The Google-Safety user agent handles abuse-specific crawling, such as malware discovery for publicly posted links on Google properties.

Operator and family: Google Type: Security

Verified robots.txt: No Block: No Verified Bot

Google-Structured-Data-Testing Safe

Google Schema Markup Testing Tool

The Google Schema Markup Testing Tool bot, now part of the Rich Results Test, crawls pages to validate their structured data. This helps webmasters check if their schema markup is correctly.

Operator and family: Google Type: Search Purpose: Indexing

Verified robots.txt: No Block: No Verified Bot

Googlebot-IA Safe

Google Scholar

Google Scholar uses a bot to crawl and index scholarly literature from academic publishers, repositories, and university websites. This populates its academic search engine.

Operator and family: Google Type: Search Purpose: Indexing

Verified robots.txt: No Block: Depends Verified Bot

Google-Site-Verification Safe

Google Site Verification

Google Site Verification is a security scanner from Google used for security checks, malware review, or abuse prevention.

Operator and family: Google Type: Security

Verified robots.txt: No Block: No Verified Bot

Storebot-Google Neutral

Google StoreBot

The Google StoreBot is a search-engine-based program that automatically 'crawls' through web pages to gather and analyse data. Google uses crawlers that go through product pages and.

Operator and family: Google Type: Ecommerce

Verified robots.txt: No Block: Depends Verified Bot

Google-Trust-Services Safe

Google Trust Services (DCV Check)

Google Trust Services (DCV Check) is a security scanner from Google used for security checks, malware review, or abuse prevention.

Operator and family: Google Type: Security

Verified robots.txt: No Block: No Verified Bot

Googlebot-Video Safe

Google Videos

The Google Videos bot is the search engine crawler for Google Video Search.

Operator and family: Google Type: Search Purpose: Indexing

Verified robots.txt: No Block: Depends Verified Bot

Google-AdWords-Express Neutral

Google-AdWords-Express

Google-AdWords-Express is a bot for a Google Ads product aimed at small businesses. It crawls advertiser websites to assist with ad creation and to verify site information.

Operator and family: Google Type: Search Purpose: Indexing

Verified robots.txt: No Block: Depends Verified Bot

Google-Adwords-Instant Neutral

Google-Adwords-Instant

Google-Adwords-Instant is a bot connected to the Google Ads platform. It visits advertiser landing pages to perform verification and quality checks.

Operator and family: Google Type: Security

Verified robots.txt: No Block: Depends Verified Bot

Google-Agent Neutral

Google-Agent

Google-Agent is an AI fetcher from Google used for AI-related crawling, retrieval, search, or assistant activity.

Operator and family: Google Type: Ai Purpose: Ai Assistant

Verified robots.txt: No Block: Depends Official Documented

Google-CloudVertexBot Safe

Google-CloudVertexBot

Google-CloudVertexBot is an AI crawler from Google used for AI-related crawling, retrieval, search, or assistant activity.

Operator and family: Google Type: Ai Purpose: Ai Crawl

Verified robots.txt: Yes Block: Depends Official Documented

Google-CWS Neutral

Google-CWS

Google-CWS is a search crawler from Google used for search indexing, page discovery, or content rendering.

Operator and family: Google Type: Search

Verified robots.txt: No Block: Depends Official Documented

Google-Display-Ads-Bot Neutral

Google-Display-Ads-Bot

Google AdSense uses a crawler called Google-Display-Ads-Bot to verify your site when you add a site to AdSense.

Operator and family: Google Type: Search Purpose: Indexing

Verified robots.txt: Yes Block: Depends Verified Bot

Google-Extended Safe

Google-Extended

Google-Extended is an AI control token from Google used for AI-related crawling, retrieval, search, or assistant activity.

Operator and family: Google Type: Ai Purpose: Ai Assistant

Verified robots.txt: Yes Block: Depends Official Documented

Google-NotebookLM Neutral

Google-NotebookLM

Google-NotebookLM is an AI fetcher from Google used for AI-related crawling, retrieval, search, or assistant activity.

Operator and family: Google Type: Ai Purpose: Ai Assistant

Verified robots.txt: No Block: Depends Official Documented

Google-Pinpoint Safe

Google-Pinpoint

Google-Pinpoint is a search crawler from Google used for search indexing, page discovery, or content rendering.

Operator and family: Google Type: Search

Verified robots.txt: No Block: Depends Official Documented

GoogleAgent-Mariner Neutral

GoogleAgent-Mariner

Claimed Google Project Mariner user-agent token for agentic browser activity.

Operator: Google Family: Google Mariner Type: Ai Purpose: Agentic Browsing

Not verified robots.txt: Unknown Block: Depends Observed

Googlebot Safe

Googlebot

Googlebot is Google Search’s web crawler for discovering, rendering, and indexing web pages.

Operator and family: Google Type: Search Purpose: Indexing

Verified robots.txt: Yes Block: Depends Official Documented

Googlebot-News Safe

Googlebot-News

Googlebot-News is a search crawler from Google used for search indexing, page discovery, or content rendering.

Operator and family: Google Type: Search Purpose: Indexing

Verified robots.txt: Yes Block: Depends Official Documented

GoogleMessages Safe

GoogleMessages

GoogleMessages is a preview or prefetch bot from Google used to fetch pages for previews, unfurls, or user-triggered rendering.

Operator and family: Google Type: Preview

Verified robots.txt: No Block: Depends Official Documented

GoogleOther Neutral

GoogleOther

Generic crawler that may be used by various product teams for fetching publicly accessible content from sites. For example, it may be used for one-off crawls for internal research and.

Operator and family: Google Type: Search Purpose: Indexing

Verified robots.txt: Yes Block: Depends Official Documented

GoogleOther-Image Neutral

GoogleOther-Image

GoogleOther-Image is an AI crawler from Google used for AI-related crawling, retrieval, search, or assistant activity.

Operator and family: Google Type: Ai Purpose: Ai Training

Verified robots.txt: Yes Block: Depends Official Documented