The most important thing to understand about Google-Extended is what it leaves untouched. Adding it to robots.txt tells Google not to use your pages to train future Gemini models or to ground its consumer AI apps, and it carries no search-ranking penalty for doing so. But the AI feature actually taking publisher clicks - the AI Overview and AI Mode answer sitting above the blue links - keeps drawing on your content regardless, because Google builds those answers from the same index Googlebot fills for ordinary search. Google-Extended controls one narrow use of your content and not the one that is replacing your traffic.
What is Google-Extended, exactly?
Google-Extended is a robots.txt token that Google introduced in September 2023 to let site owners control whether their content is used to train Google's generative AI models and to ground answers in products like the Gemini apps and Vertex AI. It is not a separate crawler and it has no user agent of its own. Google's existing crawlers, chiefly Googlebot, still fetch your pages exactly as before; the token simply governs what Google is permitted to do with what it has already collected.
That distinction matters because it changes what blocking actually achieves. When you disallow GPTBot or ClaudeBot, you stop a specific bot from fetching your pages at all. When you disallow Google-Extended, nothing stops crawling - Google keeps reading your site for Search - and you are only withdrawing permission for the training and grounding use. You will not see Google-Extended in your server logs as a distinct visitor, because there is no distinct visitor to see.
The upside Google offers in return is clean separation from Search. Google states plainly that using Google-Extended has no effect on a site's inclusion or ranking in Google Search. For a publisher that does not want its work training a rival's model, that is a real and cost-free control - as far as it goes.
What does Google-Extended not control?
Google-Extended does not remove you from AI Overviews or AI Mode. This is the point most publishers get wrong, and it is the point that matters most in 2026. AI Overviews and AI Mode are treated by Google as features of Search, not as separate AI products, so they are powered by the ordinary search index that Googlebot builds - the same index that decides whether you rank at all. Blocking Google-Extended stops Gemini training. It does nothing to the summary that answers the user's question at the top of the results page and sends the click nowhere.
The only existing lever that suppresses your content inside an AI Overview is the nosnippet directive, and nosnippet is a blunt instrument: it also strips your normal search snippet, the very text that persuades a human to click through. So the practical choice Google has left publishers with is stark. You can appear in Search with a snippet and accept that your content also feeds AI Overviews, or you can use nosnippet to stay out of the AI answer and lose your snippet in regular results at the same time. There has been no middle setting that says "index me for Search, but do not fold me into the AI answer."
That gap is the whole story. Google-Extended governs the input to training. It says nothing about the output that competes with your homepage.
How do you add Google-Extended to robots.txt?
You add Google-Extended the same way you would any other directive, by naming the token and disallowing the paths you want withheld. To withhold your entire site from Gemini training and grounding, the rule is:
User-agent: Google-Extended
Disallow: /
To withhold only part of a site, replace the slash with the directory you want to protect, for example Disallow: /premium/. Because Google-Extended is a control token rather than a fetcher, the rule changes permissions rather than blocking a request, and, to repeat the one fact worth repeating, it will not move your Search rankings. If you want your content in Search but out of Gemini's training set, this is the directive that does it, and it is the only one that does it without collateral damage to your search visibility.
Why is Google different from OpenAI and Anthropic here?
OpenAI and Anthropic each run a fleet of clearly named bots with independent robots.txt controls, so a publisher can allow one use and refuse another. OpenAI separates GPTBot for training, OAI-SearchBot for search indexing and ChatGPT-User for live user-initiated fetches. Anthropic separates ClaudeBot for training, Claude-SearchBot for indexing and Claude-User for live fetches. In both cases you can, for instance, block training while still allowing the live-answer read that might cite you. Google's architecture does not offer that seam. One Googlebot serves classic Search and the AI surfaces built on top of it, and Google-Extended sits beside it as a use-permission toggle rather than as a second crawler you can independently gate.
That fusion is why Google is simultaneously the hardest AI reader to control and, by one measure, the least extractive. Cloudflare's network data through 2026 puts Googlebot as the single largest AI-adjacent crawler, around 27 per cent of that traffic, well ahead of GPTBot and ClaudeBot. Yet Google's crawl-to-refer ratio - pages crawled per visitor sent back - has stayed close to 5 to 1, the best of the major operators, because Google still returns some clicks through the results page. Anthropic's ratio, by contrast, has run into the tens of thousands to one. Google reads more than anyone and, for now, returns more than anyone. Google-Extended does not change either number; it only decides whether the reading also trains Gemini.
What is changing with Google's opt-out controls in 2026?
The inadequacy of Google-Extended is now official. On 28 January 2026 Google acknowledged that its current tools, Google-Extended and nosnippet, do not give publishers the granular control they need to manage how their content appears in AI-generated search features separately from ordinary listings, and said it was exploring new controls. The pressure came from regulators, principally the UK's Competition and Markets Authority, which proposed measures requiring Google to let publishers opt out of AI Overviews and AI Mode without a penalty to their Search rankings.
In March 2026 Google agreed in principle to provide such an opt-out, while warning that building it is, in the words of its EMEA news partnerships lead, a "huge engineering project" - a direct admission that separating the AI answer from the search index is difficult precisely because the two share the same crawl and the same index. Publisher scepticism remains high. An opt-out that Google itself frames as hard to build, and that arrives only under regulatory duress, is not yet a control publishers can plan around. For now, Google-Extended remains what it has always been: a training toggle, not an AI-Overviews switch.
What should publishers actually do about Google-Extended?
Treat Google-Extended as a narrow, worthwhile, but incomplete measure. If you do not want your work training Gemini and you do not depend on grounding traffic from Google's AI apps, disallowing Google-Extended is a sensible, penalty-free step, and there is little reason not to take it. Just do not mistake it for a solution to the problem that is actually costing you money. It does not reduce the AI Overview's reach, it does not restore the click the AI answer absorbed, and it does not put a price on the read.
That is the deeper issue Google-Extended exposes. Blocking a training token withholds permission, but permission was never where the value leaked. The value leaks when your content is read to compose an answer the user never leaves the results page to verify. You cannot block your way to compensation with Google, because the reading you would most want to charge for is the reading fused into Search itself. This is the case for moving monetisation to the read rather than the referral. blankspace operates at the CDN edge, where the AI request is visible as it happens, so publishers can observe and monetise the AI-answer read directly instead of relying on a robots.txt token that Google's own architecture makes toothless against its most important AI surface. Set Google-Extended for what it is good for, and treat the answer layer as a separate, revenue-bearing problem.
Frequently asked questions
Does blocking Google-Extended hurt my Google Search rankings?
No. Google states that using Google-Extended has no effect on a site's inclusion or ranking in Google Search. It is the one control that cleanly separates AI-training permission from search visibility, so disallowing it withdraws your content from Gemini training and grounding without any Search penalty. This is the specific reason Google created a distinct token rather than folding the choice into an existing crawler directive.
Does Google-Extended stop my content appearing in AI Overviews?
No. AI Overviews and AI Mode are features of Google Search, drawn from the index that Googlebot builds, so Google-Extended does not touch them. The only existing directive that suppresses your content inside an AI Overview is nosnippet, and it also removes your ordinary search snippet, which usually costs more traffic than it saves. Until Google ships its promised dedicated opt-out, there is no clean way to stay in Search but out of the AI answer.
Is Google-Extended a crawler I can see in my server logs?
No. Google-Extended is a robots.txt control token, not a fetcher, and it has no user agent of its own. The actual crawling is done by Google's existing crawlers, chiefly Googlebot, and the token only governs whether the content they collect may be used to train and ground Google's generative AI. If you are looking for a "Google-Extended" line in your access logs, you will not find one.
What is the difference between Google-Extended and Googlebot?
Googlebot is the crawler that fetches your pages for Google Search and, by extension, for the AI surfaces built on the search index. Google-Extended is a permission token that decides whether that already-crawled content can also train Gemini and Vertex AI. You can restrict Google-Extended freely, but you cannot restrict Googlebot without dropping out of Google Search altogether, which is why blocking AI use of your content is far harder with Google than with OpenAI or Anthropic.
Should small publishers block Google-Extended?
For most small publishers, disallowing Google-Extended is a low-risk default: it stops your content training a model you are unlikely to be paid for, and it costs you nothing in Search. The exception is a publisher that gets meaningful referral traffic from Gemini grounding and wants to stay eligible for it. Either way, blocking the token is a decision about training, not about traffic - the traffic problem created by AI Overviews needs a different answer, because Google-Extended cannot reach it.