How to configure robots.txt for GPTBot, OAI-SearchBot and ChatGPT-User

OpenAI documents, on the platform's official bots page, four user-agents with distinct functions: GPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot. The problem is that many people treat the first three as if they were the same thing and write a generic User-agent: * rule in robots.txt, blocking or allowing everything at once. That throws away half the control OpenAI itself gives the webmaster.

The documentation is explicit: "each configuration is independent of the others," and a site can allow OAI-SearchBot to appear in search results while blocking GPTBot to prevent crawled content from being used in training. That's the fine-tuning this guide walks through, from the raw log to the test in ChatGPT Search.

Three bots, three different decisions

Before touching robots.txt, it's worth pinning down what each agent does, because the decision to block or allow changes depending on the goal.

OpenAI itself warns that, if a site allows both the training and search bots, it "may use the result of a single crawl for both uses, to avoid duplicate crawling." In other words: allowing both doesn't double the load on the server, but blocking only one requires separate, explicit rules, because a generic rule treats all agents the same way.

Step 1: audit the logs before touching any rule

The first mistake is editing robots.txt without knowing whether these bots are already accessing (or not) the site. Run a count per user-agent on the server's access log:

grep -ic "GPTBot" /var/log/nginx/access.log
grep -ic "OAI-SearchBot" /var/log/nginx/access.log
grep -ic "ChatGPT-User" /var/log/nginx/access.log

If all three return zero, the site has probably never been crawled by any of them, and the priority is to allow OAI-SearchBot before thinking about blocking anything. If GPTBot already shows up in volume and you want to prevent training, confirm the requests really come from OpenAI, and not from someone spoofing the User-Agent header:

grep "GPTBot" /var/log/nginx/access.log | awk '{print $1}' | sort -u > ips_gptbot.txt
curl -s https://openai.com/gptbot.json | jq -r '.prefixes[].ipv4Prefix' > faixas_oficiais.txt

With both lists in hand, cross-check the IPs from your log against the published CIDR prefixes (openai.com/gptbot.json, openai.com/searchbot.json and openai.com/chatgpt-user.json, one JSON file per bot). A request with User-Agent: GPTBot coming from outside those ranges is, at minimum, suspicious and shouldn't inform your decision to block or allow.

Step 2: write separate rules in robots.txt

With the logs audited, robots.txt looks like this for those who want to appear in ChatGPT search but don't want to feed model training:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

Notice there's no User-agent: * block in this example. If you already have a generic catch-all in the file (User-agent: * followed by Disallow: /admin, for example), it still applies to any bot without its own block, but the specific blocks for GPTBot, OAI-SearchBot and ChatGPT-User always take priority over the generic one for those three agents. That's how you block training without blocking search: the more specific rule wins.

It's worth remembering three details the documentation makes explicit, which change the strategy for those who rely on automation:

• ChatGPT-User only acts when a user asks for something specific inside ChatGPT; OpenAI warns that, since it's an action triggered by a person, "robots.txt rules may not apply" to it the same way they apply to an automated crawler.
• OAI-SearchBot is the only agent that decides whether the site appears in search responses; blocking ChatGPT-User doesn't remove the site from search, and allowing ChatGPT-User doesn't guarantee it will appear there.
• If the site also runs ads on ChatGPT, there's still OAI-AdsBot to consider, which only visits pages submitted as ads and, according to the docs, never feeds foundation model training.

Step 3: validate by testing real prompts in ChatGPT Search

Editing the file isn't the end of the work: you need to confirm the change actually affected what appears in ChatGPT's responses. OpenAI warns that, for OAI-SearchBot, "it can take about 24 hours, from the moment a site's robots.txt is updated, for the systems to adjust." Plan the test in two rounds, with that interval in between.

1. Before editing the file, choose three to five prompts a real user would ask, which by subject should pull up a page from your site as a source. Run each one in ChatGPT with search enabled and record whether the domain appears as a citation or link.
2. Publish the new robots.txt rules and wait at least 24 hours, the timeframe OpenAI itself indicates for search reindexing.
3. Run the same prompts again, under the same conditions, and compare: if you allowed OAI-SearchBot, the domain is expected to start appearing (or appear again) in the responses; if you blocked it, the opposite is expected.
4. In parallel, go back to the server logs and confirm GPTBot stopped hitting the site (or that OAI-SearchBot started appearing, if that was the allowance made). A crawler that ignores a Disallow after 24h is a sign of an outdated robots.txt cache on your CDN, not bad faith from the bot.

The two-sided test (server log + prompt in the product) is what closes the loop: the log shows whether the crawler obeyed the rule, the prompt shows whether the combined effect (allowed crawling, search indexing, display in the response) reached the end user.

Common mistakes that trip up this configuration

Some stumbles show up frequently in this type of configuration, and it's worth checking each one before considering the task done:

• Leaving a User-agent: * with Disallow: / at the top of the file and assuming it doesn't affect OpenAI's three bots: it does, unless there's a specific per-agent block afterward.
• Confusing "I blocked GPTBot" with "I blocked search appearance": they're different systems, controlled by different agents.
• Validating the rule just by looking at the rendered robots.txt, without checking the server log after a few days, to confirm the bot actually stopped (or started) accessing.
• Ignoring ChatGPT-User by assuming it always obeys robots.txt the same way an automated crawler does; the docs themselves warn that, being a user action, that may not hold.

What's still left open

OpenAI's documentation is clear about what each bot does going forward, but it doesn't say what happens to content already crawled by GPTBot before a Disallow takes effect. Blocking today prevents new crawling; there's no guarantee, on the page itself, about data already used in previous training runs. For those who measure GEO (generative search visibility) as a traffic metric, the logical next step after this configuration is to repeat the step 3 test periodically, because OpenAI may update how OAI-SearchBot weighs content without necessarily announcing the change.