OpenAI separates GPTBot, OAI-SearchBot and ChatGPT-User: how to configure robots.txt without losing citation in ChatGPT Search

OpenAI's crawler documentation, available at platform.openai.com/docs/bots, makes a point that still confuses a lot of people who handle technical SEO: there is more than one bot, and each one does something different with the site's content. Blocking one doesn't block the other, and it's exactly this independence that growth teams usually ignore when they write a generic robots.txt for "OpenAI's AI."

The most common mistake is to treat OpenAI as a single block: either allow everything or block everything. In practice, OpenAI documents four user-agents with distinct functions, and at least three of them matter directly to anyone thinking about GEO (generative engine optimization) in Brazil.

Four bots, four functions

The table below summarizes what each crawler does, according to OpenAI's own documentation:

OpenAI itself states this explicitly: "Each setting is independent of the others." In other words, it's possible to allow OAI-SearchBot while at the same time denying GPTBot, and that's exactly the combination that makes sense for most sites that want to appear in ChatGPT answers without authorizing their text to become training data.

The robots.txt that SEO teams still get wrong (before)

A common pattern, copied from generic "AI blocking" templates, treats all of OpenAI's agents as a single block:

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

This block blocks everything at once. The side effect is that the site stops appearing as a cited source in ChatGPT Search answers, because OAI-SearchBot never even gets to index the content. For a site that lives off discovery traffic, this means unnecessarily giving up a distribution surface, since training and search indexing didn't need to be tied together.

The configuration that separates the two things (after)

The fix that solves the problem is simple to write, but requires understanding each bot's role before applying it:

# Blocks content from being used to train foundation models
User-agent: GPTBot
Disallow: /

# Allows appearing in ChatGPT Search answers
User-agent: OAI-SearchBot
Allow: /

# Allows user actions (e.g., GPT Actions) to access the page
User-agent: ChatGPT-User
Allow: /

With this rule, the site stays out of OpenAI's model training pipeline, but remains eligible to be cited when someone searches on ChatGPT. OpenAI's own documentation confirms that, if both bots are allowed, OpenAI may reuse the result of a single crawl for both purposes, to avoid duplicate crawling: this doesn't change the logic that each Disallow acts in isolation.

The ChatGPT-User catch

The point that most trips up an inattentive SEO manager is ChatGPT-User. It is not an automatic crawler: it's triggered when a user, inside a ChatGPT conversation or using a Custom GPT, asks the tool to visit a specific page or run a GPT Action. The documentation is direct in stating that, because it's a user-triggered action, robots.txt rules may simply not apply.

This has two practical consequences. First, blocking ChatGPT-User doesn't necessarily prevent the one-off access a user requests during a conversation; it's an etiquette control, not a technical guarantee of blocking. Second, and more important for GEO: ChatGPT-User is not used to decide whether content appears in ChatGPT Search. Anyone who wants to control search presence needs to adjust OAI-SearchBot, not ChatGPT-User (confusing the two wastes configuration effort with no effect on the real goal).

IP verification: avoiding fake bots

OpenAI publishes each crawler's IP ranges in its own JSON files: openai.com/searchbot.json for OAI-SearchBot, openai.com/gptbot.json for GPTBot, openai.com/chatgpt-user.json for ChatGPT-User, and openai.com/adsbot.json for OAI-AdsBot. This matters because any bot can forge the User-Agent header in a request. Validating the source IP against these lists, before applying cache rules or serving differentiated content, is the way to confirm that the visit really comes from OpenAI and not from a scraper impersonating it.

What to measure after the adjustment

Anyone who applies this separation can validate the effect in two concrete ways: monitoring OAI-SearchBot requests in the server log (confirming that they started occurring, or continued, after the Allow) and checking, through manual searches on ChatGPT, whether the domain starts appearing as a cited source for questions related to the site's content. The documentation itself warns that changes to robots.txt take about 24 hours to reflect in ChatGPT's search results, so the before-and-after test only makes sense with that waiting window: comparing the log the day after the change tends to show a crawl count that's still lagging.

When it's not worth changing anything

For sites that have no interest whatsoever in appearing as a source in generative AI answers (strictly internal content, pages behind a login, or editorial outlets that already block all indexing as a matter of policy), the distinction between GPTBot, OAI-SearchBot, and ChatGPT-User is irrelevant: total blocking remains the coherent choice. Granularity only pays off for those who depend on discovery and referral traffic, the typical scenario for news portals, technical documentation, and product blogs that compete for citation inside ChatGPT Search.