Google details what Google-Extended, GoogleOther, and Googlebot control in robots.txt
Google Search Central's documentation, updated on October 30, 2026, clearly separates what each crawler user-agent does. The most common confusion, between Googlebot and Google-Extended, can cost dearly anyone who configures robots.txt without understanding the difference.

- Reading
- 5 min
- Tables
- 1
- Code
- 1
A documentação do Google Search Central... [translation below]
Google Search Central's documentation, updated on October 30, 2026, clearly separates what each crawler user-agent does. The most common confusion, between Googlebot and Google-Extended, can cost dearly anyone who configures robots.txt without understanding the difference.
Editor's note: the official documentation cited in this article never mentions the AI Overviews feature. The specific relationship drawn in the section below, that blocking Google-Extended alone does not remove a site from AI Overviews and that opting out of that feature entirely would require blocking Googlebot altogether, is a logical inference by the author based on how Search works, not a verifiable claim in the official source presented. The rest of the user-agent mapping in the article was confirmed in Google's documentation.
Google Search Central's documentation, updated on October 30, 2026 according to the document's own footer, consolidates into a single table the user-agents Google uses to crawl the web. The list itself isn't new, but the level of detail about which product each token affects is what matters now for anyone deciding what goes in or out of robots.txt. That applies to both traditional SEO and GEO: optimizing for generative AI.
The central point for anyone touching robots.txt this week is simple to state and easy to get wrong in practice: not every Google AI crawler is the same thing, and blocking the wrong one doesn't produce the effect the team imagines. The table below summarizes what the documentation confirms, product by product.
| User-agent (robots.txt) | What it affects if blocked |
|---|---|
Googlebot | Full Search (Discover, Images, Video, News) |
Googlebot-Image | Google Images, Discover, video mode |
Googlebot-News | Google News (news.google.com and the app) |
Storebot-Google | Google Shopping and the Shopping tab in Search |
Google-InspectionTool | Only testing tools (Rich Result Test, URL Inspection) |
GoogleOther | No specific product; Google's internal research |
GoogleOther-Image / GoogleOther-Video | Same as GoogleOther, but for images and videos |
Google-CloudVertexBot | Vertex AI agents requested by the site's own owner |
Google-Extended | Training Gemini models and grounding (Gemini Apps, Vertex AI API for Gemini) |
Google-Extended doesn't remove your site from Search, nor from AI Overviews
Here's the confusion that shows up most often in poorly informed SEO checklists: assuming that blocking Google-Extended removes content from AI Overviews. The documentation is direct in stating that this token controls only two things: whether the crawled content can train future generations of Gemini models, and whether it can be used in grounding, when a model's answer pulls context from the search index at prompt time to improve accuracy.
The source text closes the doubt without leaving room for ambiguity: Google-Extended doesn't affect a site's inclusion in Search, nor does it work as a ranking signal. Since AI Overviews is a feature within Search itself, built on the index that Googlebot feeds, blocking Google-Extended alone doesn't stop a snippet of the site from appearing summarized there. Anyone who wants to opt out of AI Overviews entirely has only one real lever, which is blocking Googlebot, and that costs the entire indexing, not just the AI feature.
GoogleOther and Google-CloudVertexBot: blocking breaks nothing visible
The documentation describes GoogleOther in terms worth quoting directly, because they explain why it often shows up in server logs without producing any noticeable effect when it's allowed or blocked:
GoogleOther is the generic crawler that may be used by various product teams for fetching publicly accessible content from sites.
Google Search Central, official documentation
In practice, this means GoogleOther is used for internal research and one-off experiments by Google teams, without ties to a fixed product. The same reasoning applies to Google-CloudVertexBot: it only runs when the site's own owner has hired an agent on Vertex AI, and the documentation confirms it has no effect on Search or any other product. Blocking both in robots.txt is a low-risk decision for anyone who only cares about Search and AI Overviews.
The before/after test checklist
Before editing robots.txt in production, it's worth running a short sequence that separates intention from actual effect:
- Audit the current
robots.txtline by line and mark which token is in each rule, without assuming that "blocking AI" is a single rule. - Separate the objectives: keeping indexing in Search (
Googlebot) is one decision; weighing in on Gemini training (Google-Extended) is another; neither touches Shopping (Storebot-Google) or Vertex agents (Google-CloudVertexBot). - Review the syntax of the rules before publishing, and use the URL Inspection tool in Search Console to confirm that
Googlebotcan still see the pages that matter. - Cross-check the server logs filtering by user-agent; the documentation recommends using a wildcard in the
Chrome/W.X.Y.Zsegment, since that version number changes with the Chromium thatGooglebotuses internally. - Compare indexing coverage in Search Console before and after the change, to confirm that only the expected behavior (AI training, for example) was affected, with no drop in indexed pages.
The step that's missing, and that no dashboard solves today, is measuring the effect of Google-Extended in isolation: in our reading, there's no public report confirming whether the block excluded the content from the next Gemini training cycle, since that point isn't addressed by the official documentation. Confirmation depends on a policy promise declared by the company, not on auditable data in Search Console.
Ready-to-adapt robots.txt blocks
For anyone who wants to keep full indexing in Search (including AI Overviews), but opt out of Gemini training and grounding, the minimal combination is:
User-agent: Googlebot
Allow: /
User-agent: Google-Extended
Disallow: /
User-agent: GoogleOther
Disallow: /This block allows the crawler that sustains Search, blocks the use of content in Gemini, and removes traffic from internal research with no defined product. Teams that sell on Google Shopping need to add their own rule for Storebot-Google, since it's not affected by any of the three above; the same goes for anyone using Vertex AI agents and wanting to control them via Google-CloudVertexBot.
What remains open
The documentation resolves the ambiguity of "which token does what," but it doesn't resolve the opacity of what happens afterward: there's no way to verify, from the outside, whether a Google-Extended block made today truly prevents a future version of Gemini from having been trained on content crawled before the change. For GEO, that means treating robots.txt as declared policy, not as an auditable technical guarantee, and repeating the log-and-Search-Console checklist whenever Google updates this list again.
Translated from the Brazilian Portuguese original · Read the original
Search Console Has No Filter to Isolate Clicks From AI Overviews and AI Mode
The Google Search Central documentation, updated in December 2025, confirms that AI Overviews and AI Mode traffic falls under the Web search type in the Performance report, with no segmentation of its own. Here's what you can, and can't, measure.






