Googlebot

Vendor: Google
Type: Search
JavaScript rendering: yes
Honors robots.txt: yes


Vendor	Google
Type	Traditional search crawler
robots.txt token	`Googlebot`
JavaScript rendering	Yes — Web Rendering Service (Chrome-based, near-real-time)
Honors robots.txt	Yes
Vendor docs	developers.google.com/crawling/docs/crawlers-fetchers

User-Agent strings

Desktop:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/131.0.6778.85 Safari/537.36

Smartphone (primary index since 2021):

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.6778.85 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Image / video / news variants append -Image, -Video, -News to the token.

Purpose

The crawler behind Google Search. Discovers new URLs, re-fetches known URLs to refresh the index, and feeds the rendering pipeline that produces the indexed snapshot of every page. Renders JavaScript via Chrome — the "WRS" rendering tier runs the same Chromium engine end-users have, on roughly the same release cadence.

Network identity

Reverse-DNS hostname pattern: .googlebot.com and .google.com
Authoritative IP list: published at

https://developers.google.com/static/search/apis/ipranges/googlebot.json (updated regularly; verify before block decisions)

How to verify

# 1) Reverse-DNS lookup on the request IP must resolve to .googlebot.com
# 2) Forward-DNS lookup on that hostname must resolve back to the same IP
# 3) Then check the User-Agent.

UA alone is spoofable in seconds. Always pair UA matching with rDNS+fDNS verification or with an IP-range match against the official JSON.

In our audit

In v1 (top-30 general), Googlebot reached 23/30 sites successfully. Wikipedia, TikTok, X, cloud.microsoft, and ebay specifically returned 403 to the Googlebot UA from our non-Google IP — the correct behavior for sites that validate bot identity by reverse-DNS.

In v2 (top-100 e-commerce), Googlebot received fully-pre-rendered HTML from baidu (700×), amazon.co.uk (24×), coupang (5×), shopping.yahoo.co.jp (7×), and uniqlo (3×) — the "dynamic rendering for trusted crawlers" pattern.

How to allow / block

User-agent: Googlebot
Disallow: /private/
Allow: /

Blocking Googlebot is rarely defensible — it removes the site from Google Search. If you genuinely need to opt out, use noindex headers on sensitive paths instead.

Quirks

Googlebot's WRS has a "render cap" — pages that depend on user

interaction (click, scroll, hover) to reveal content will not have that content indexed. Use intersection-observer-style lazy-load only for below-the-fold.

The smartphone profile is the primary index. If your responsive CSS

serves different content to mobile, that's what Google sees.

Bot reference catalog