SEARCH · Google

Googlebot

The reference search crawler

The crawler behind Google Search. Discovers new URLs, refreshes the index, and runs JavaScript via the Web Rendering Service (Chromium-based, near-real-time as of 2024).

Specs

VendorGoogle
CategorySEARCH
robots.txt tokenGooglebot
Renders JavaScriptRENDERS JS
Honors robots.txtyes
Reverse-DNS pattern*.googlebot.com / *.google.com
IP-range sourcehttps://developers.google.com/static/search/apis/ipranges/googlebot.json

User-Agent strings

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.6778.85 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/131.0.6778.85 Safari/537.36

Considerations

  • Smartphone profile is the primary index since 2021. Responsive sites must render correctly under that UA.
  • Verify with reverse-DNS + forward-DNS. UA-only matching is spoofable.
  • Wikipedia, TikTok, eBay and several other sites in our May 2026 audit returned HTTP 403 to a Googlebot UA from a non-Google IP — that is correct, healthy behavior on their part.

robots.txt recipe

User-agent: Googlebot
Allow: /
Disallow: /admin/
Related bots: Google-Extended · Bingbot · YandexBot

Sources: Google · Crawlers overview · Googlebot IP ranges

← Back to directory