MartechARTICLE

Google details what Google-Extended, GoogleOther, and Googlebot control in robots.txt

Google Search Central's documentation, updated on October 30, 2026, clearly separates what each crawler user-agent does. The most common confusion, between Googlebot and Google-Extended, can cost dearly anyone who configures robots.txt without understanding the difference.

Google details what Google-Extended, GoogleOther, and Googlebot control in robots.txt
Image: Sabrina Santos
Article · Martech
Reading
5 min
Tables
1
Code
1

A documentação do Google Search Central... [translation below]

Google Search Central's documentation, updated on October 30, 2026, clearly separates what each crawler user-agent does. The most common confusion, between Googlebot and Google-Extended, can cost dearly anyone who configures robots.txt without understanding the difference.

Editor's note: the official documentation cited in this article never mentions the AI Overviews feature. The specific relationship drawn in the section below, that blocking Google-Extended alone does not remove a site from AI Overviews and that opting out of that feature entirely would require blocking Googlebot altogether, is a logical inference by the author based on how Search works, not a verifiable claim in the official source presented. The rest of the user-agent mapping in the article was confirmed in Google's documentation.

Google Search Central's documentation, updated on October 30, 2026 according to the document's own footer, consolidates into a single table the user-agents Google uses to crawl the web. The list itself isn't new, but the level of detail about which product each token affects is what matters now for anyone deciding what goes in or out of robots.txt. That applies to both traditional SEO and GEO: optimizing for generative AI.

The central point for anyone touching robots.txt this week is simple to state and easy to get wrong in practice: not every Google AI crawler is the same thing, and blocking the wrong one doesn't produce the effect the team imagines. The table below summarizes what the documentation confirms, product by product.

User-agent (robots.txt)What it affects if blocked
GooglebotFull Search (Discover, Images, Video, News)
Googlebot-ImageGoogle Images, Discover, video mode
Googlebot-NewsGoogle News (news.google.com and the app)
Storebot-GoogleGoogle Shopping and the Shopping tab in Search
Google-InspectionToolOnly testing tools (Rich Result Test, URL Inspection)
GoogleOtherNo specific product; Google's internal research
GoogleOther-Image / GoogleOther-VideoSame as GoogleOther, but for images and videos
Google-CloudVertexBotVertex AI agents requested by the site's own owner
Google-ExtendedTraining Gemini models and grounding (Gemini Apps, Vertex AI API for Gemini)

Google-Extended doesn't remove your site from Search, nor from AI Overviews

Here's the confusion that shows up most often in poorly informed SEO checklists: assuming that blocking Google-Extended removes content from AI Overviews. The documentation is direct in stating that this token controls only two things: whether the crawled content can train future generations of Gemini models, and whether it can be used in grounding, when a model's answer pulls context from the search index at prompt time to improve accuracy.

The source text closes the doubt without leaving room for ambiguity: Google-Extended doesn't affect a site's inclusion in Search, nor does it work as a ranking signal. Since AI Overviews is a feature within Search itself, built on the index that Googlebot feeds, blocking Google-Extended alone doesn't stop a snippet of the site from appearing summarized there. Anyone who wants to opt out of AI Overviews entirely has only one real lever, which is blocking Googlebot, and that costs the entire indexing, not just the AI feature.

GoogleOther and Google-CloudVertexBot: blocking breaks nothing visible

The documentation describes GoogleOther in terms worth quoting directly, because they explain why it often shows up in server logs without producing any noticeable effect when it's allowed or blocked:

GoogleOther is the generic crawler that may be used by various product teams for fetching publicly accessible content from sites.

Google Search Central, official documentation

In practice, this means GoogleOther is used for internal research and one-off experiments by Google teams, without ties to a fixed product. The same reasoning applies to Google-CloudVertexBot: it only runs when the site's own owner has hired an agent on Vertex AI, and the documentation confirms it has no effect on Search or any other product. Blocking both in robots.txt is a low-risk decision for anyone who only cares about Search and AI Overviews.

The before/after test checklist

Before editing robots.txt in production, it's worth running a short sequence that separates intention from actual effect:

  1. Audit the current robots.txt line by line and mark which token is in each rule, without assuming that "blocking AI" is a single rule.
  2. Separate the objectives: keeping indexing in Search (Googlebot) is one decision; weighing in on Gemini training (Google-Extended) is another; neither touches Shopping (Storebot-Google) or Vertex agents (Google-CloudVertexBot).
  3. Review the syntax of the rules before publishing, and use the URL Inspection tool in Search Console to confirm that Googlebot can still see the pages that matter.
  4. Cross-check the server logs filtering by user-agent; the documentation recommends using a wildcard in the Chrome/W.X.Y.Z segment, since that version number changes with the Chromium that Googlebot uses internally.
  5. Compare indexing coverage in Search Console before and after the change, to confirm that only the expected behavior (AI training, for example) was affected, with no drop in indexed pages.

The step that's missing, and that no dashboard solves today, is measuring the effect of Google-Extended in isolation: in our reading, there's no public report confirming whether the block excluded the content from the next Gemini training cycle, since that point isn't addressed by the official documentation. Confirmation depends on a policy promise declared by the company, not on auditable data in Search Console.

Ready-to-adapt robots.txt blocks

For anyone who wants to keep full indexing in Search (including AI Overviews), but opt out of Gemini training and grounding, the minimal combination is:

User-agent: Googlebot
Allow: /

User-agent: Google-Extended
Disallow: /

User-agent: GoogleOther
Disallow: /

This block allows the crawler that sustains Search, blocks the use of content in Gemini, and removes traffic from internal research with no defined product. Teams that sell on Google Shopping need to add their own rule for Storebot-Google, since it's not affected by any of the three above; the same goes for anyone using Vertex AI agents and wanting to control them via Google-CloudVertexBot.

What remains open

The documentation resolves the ambiguity of "which token does what," but it doesn't resolve the opacity of what happens afterward: there's no way to verify, from the outside, whether a Google-Extended block made today truly prevents a future version of Gemini from having been trained on content crawled before the change. For GEO, that means treating robots.txt as declared policy, not as an auditable technical guarantee, and repeating the log-and-Search-Console checklist whenever Google updates this list again.

Translated from the Brazilian Portuguese original · Read the original