Google-Extended
| Vendor | |
| Type | AI training opt-out token (not a separate crawler) |
| robots.txt token | Google-Extended |
| JavaScript rendering | n/a — control directive only |
| Honors robots.txt | Yes |
| Vendor docs | developers.google.com/search/docs/crawling-indexing/overview-google-crawlers |
What it actually is
Google-Extended is a robots.txt token, not a crawler. Sites that include a Disallow: / for it opt out of having Googlebot-fetched content used to train Gemini, Vertex AI, and the rest of Google's generative AI stack — while keeping their Google Search visibility intact.
Introduced September 2023 in response to publisher concerns about generative AI training opt-out. Apple's Applebot-Extended (2024) and OpenAI's allow/disallow split between GPTBot and ChatGPT-User follow the same pattern.
How to use
To opt out of Gemini training but keep Google Search:
User-agent: Googlebot
Allow: /
User-agent: Google-Extended
Disallow: /In our audit
Did not appear as a UA in any of our test traffic (it's a directive, not a fetcher). We did not test sites' robots.txt files for the presence of Google-Extended directives — that would be a useful companion audit.