Bot intelligence record

Googlebot

Usually allow

Use the Googlebot identifier to separate Google search indexing or content discovery traffic from normal visitor requests in server logs.

Search Indexing Official Documented Confidence: High Verified: Yes robots.txt: Yes

User-Agent Pattern

Google
Googlebot

Robots.txt Snippet

Copy-ready 
User-agent: Googlebot Disallow: /

Handling Guidance

Depends

This bot is usually safe to allow when the request source is verified and the traffic matches your site policy.

Search indexing, content discovery, rendering, or search-result freshness checks.

Record Details

Structured data
Operator
Google
Family
Google
Type
Search
Purpose
Indexing
Identity type
Official Documented
Confidence
High
Last verified
2026-04-01
Last checked
2026-04-30
Source type
Official
Spoofing risk
User-agent strings can be spoofed. For allow-listing or low-friction rules, pair the published identifier with operator documentation or cryptographic verification when available.

Notes

Googlebot is listed in the Botcrawl directory as a search crawler from Google. The primary identifier for log review is Googlebot.

Identification

  • User-agent pattern: Googlebot
  • Family: Google
  • Type: Search
  • Kind: Crawler

Common use

Search indexing, content discovery, rendering, or search-result freshness checks.

Verification and handling

Use Google's official crawler verification guidance because user-agent strings can be spoofed.

Directory guidance marks the risk level as Safe and the blocking decision as Depends. Do not rely on the user-agent string alone because user-agent strings can be copied or spoofed.

Robots.txt handling: Yes.

Evidence and Source

  • Use Google's official crawler verification guidance because user-agent strings can be spoofed.
  • Search indexing, content discovery, rendering, or search-result freshness checks.
  • User-agent strings can be spoofed. For allow-listing or low-friction rules, pair the published identifier with operator documentation or cryptographic verification when available.

https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers

Same family: Google

ProjectShield-UrlCheck Safe

ProjectShield Url Check

ProjectShield Url Check is a security scanner from Google used for security checks, malware review, or abuse prevention.

Google Google Security Security
Verified robots.txt: No Block: No Verified Bot
GoogleOther Neutral

GoogleOther

Generic crawler that may be used by various product teams for fetching publicly accessible content from sites. For example, it may be used for one-off crawls for internal research and.

Google Google Search Indexing
Verified robots.txt: Yes Block: Depends Official Documented
GoogleOther-Image Neutral

GoogleOther-Image

GoogleOther-Image is an AI crawler from Google used for AI-related crawling, retrieval, search, or assistant activity.

Google Google Ai Ai Training
Verified robots.txt: Yes Block: Depends Official Documented
GoogleMessages Safe

GoogleMessages

GoogleMessages is a preview or prefetch bot from Google used to fetch pages for previews, unfurls, or user-triggered rendering.

Google Google Preview Preview
Verified robots.txt: No Block: Depends Official Documented

Same operator: Google

GoogleOther-Video Neutral

GoogleOther-Video

GoogleOther-Video is an AI crawler from Google used for AI-related crawling, retrieval, search, or assistant activity.

Google Google Ai Ai Training
Verified robots.txt: Yes Block: Depends Official Documented
Google-Extended Safe

Google-Extended

Google-Extended is an AI control token from Google used for AI-related crawling, retrieval, search, or assistant activity.

Google Google Ai Ai Assistant
Verified robots.txt: Yes Block: Depends Official Documented
Google-NotebookLM Neutral

Google-NotebookLM

Google-NotebookLM is an AI fetcher from Google used for AI-related crawling, retrieval, search, or assistant activity.

Google Google Ai Ai Assistant
Verified robots.txt: No Block: Depends Official Documented
Google-Agent Neutral

Google-Agent

Google-Agent is an AI fetcher from Google used for AI-related crawling, retrieval, search, or assistant activity.

Google Google Ai Ai Assistant
Verified robots.txt: No Block: Depends Official Documented

Similar purpose: Indexing

YandexWebmaster Safe

YandexWebmaster

YandexWebmaster is a search crawler from Yandex used for search indexing, page discovery, or content rendering.

Yandex Yandex Search Indexing
Verified robots.txt: Yes Block: Depends Official Documented
YandexVideoParser Safe

YandexVideoParser

YandexVideoParser is a search crawler from Yandex used for search indexing, page discovery, or content rendering.

Yandex Yandex Search Indexing
Verified robots.txt: No Block: Depends Official Documented
YandexSpravBot Safe

YandexSpravBot

YandexSpravBot is a search crawler from Yandex used for search indexing, page discovery, or content rendering.

Yandex Yandex Search Indexing
Verified robots.txt: Yes Block: Depends Official Documented
YandexVertis Safe

YandexVertis

YandexVertis is a search crawler from Yandex used for search indexing, page discovery, or content rendering.

Yandex Yandex Search Indexing
Verified robots.txt: Yes Block: Depends Official Documented

WordPress Bot Protection

Bot Blocker for WordPress

Monitor bot traffic, review live activity, and control AI crawlers, scrapers, scanners, spam bots, and fake trusted bots from one clean WordPress dashboard.

Buy Now Start Free Trial