llms.txt reaches version 2, but support from major AI models is still unconfirmed

A especificação llmstxt.org chegou à segunda versão depois de dois anos de adoção desde 2024. Para quem faz GEO, a pergunta que importa é se publicar o arquivo move tráfego de verdade ou só atende quem já publica por conta própria.

The file that promises documentation for robots

The proposal was born in 2024 and became something of a small de facto standard: thousands of sites publish a /llms.txt file at the root, or in any subfolder, such as /docs/llms.txt, covering the pages under that path. The idea is simple. Instead of forcing an AI agent to scrape an HTML page full of menus, ads, and JavaScript, the site offers a markdown summary with direct links to the relevant content, already in a lean format.

The specification, hosted at llmstxt.org, reached version 2 after two years of adoption since the original 2024 proposal. It changed little in essence, but incorporated the lessons learned from those who had already published the file: the structure became stricter, to allow reading by both LLMs and classic parsing and regex tools.

How the format works, section by section

The specification defines a fixed order of markdown blocks. Only the title is mandatory; the rest is optional, but follows a specific order when present:

1. An H1 with the project or site name (the only mandatory item).
2. A blockquote (>) with a short summary of what the site does.
3. Free-form paragraphs or lists with additional context.
4. H2 sections with lists of links in the format [name](url): optional description.

A reduced example, based on what the FastHTML project publishes in its llms.txt:

# FastHTML

> FastHTML is a Python library that combines Starlette, Uvicorn, HTMX, and fastcore's FastTags into a library for server-rendered apps.

## Docs

- [Quick start](https://fastht.ml/docs/tutorials/quickstart_for_web_devs.html.md): overview of the main features

## Optional

- [Todo list example](https://github.com/AnswerDotAI/fasthtml/blob/main/examples/adv_app.py): complete CRUD app

The section called Optional, by convention, gathers links that an agent can ignore when the context needs to be shorter. The specification also proposes publishing a markdown version of each page (page.html.md or page.md), announced via rel="alternate" type="text/markdown" in the HTML or via an HTTP Link: header, which allows adding the feature through CDN configuration without touching the pages' code.

How this fits alongside robots.txt and sitemap.xml

The specification is explicit: llms.txt does not replace the two standards that already exist at the root of any site. The three coexist, but serve different audiences and purposes.

Who publishes it, and who already generates it automatically

The adoption cited in the specification itself gives clues about who is taking the standard seriously. The documentation platform Mintlify automatically generates the llms.txt file and the .md versions for every site it hosts; GitBook, in turn, serves an llms.txt file for documentation sites published on its platform, though the specification doesn't detail automatic generation of .md versions in that case. WordPress plugins such as Yoast SEO and AIOSEO already include a generator for the file, and Wix does the same for any site created on the platform.

On the framework side, ready-made plugins exist: vitepress-plugin-llms for VitePress, docusaurus-plugin-llms for Docusaurus, a recipe for Drupal 10.3+, and even an MCP server, server-llm-txt, which lets an agent search for and look up llms.txt files across the network. Chrome's Lighthouse now audits the file's presence as part of its agentic browsing checks. OpenAI, Anthropic, and Google publish llms.txt files for their own developer documentation, a real data point cited in the specification, but one that deserves careful reading.

The blind spot: publishing is not the same as being read

Here lies the question that matters to those working in growth and technical SEO: the fact that OpenAI, Anthropic, and Google publish llms.txt for their own docs doesn't prove that ChatGPT, Claude, or Gemini read third-party llms.txt files when answering a user. These are different things. One is the company helping agents, including competitors', understand its own API; the other would be the AI product consuming a random site's file during a search or a response.

There is no confirmation, either in the specification or in public documentation from OpenAI, Anthropic, or Google, that the crawlers used for training or retrieval, such as GPTBot, ClaudeBot, or Google-Extended, treat llms.txt in any special way. The document itself is honest about this limitation:

Our expectation was that llms.txt would mainly be useful for inference rather than training, and that is how it has been used, though training runs could take advantage of the information too.

Notice the verb: expectation, not confirmation. The document describes adoption (automatic generators, Lighthouse audits, directories like llmstxt.site) but cites no official announcement from any model provider stating that its retriever consults /llms.txt before responding. For those doing GEO, this distinction between an adopted signal and a confirmed signal is the same caution that already applied to structured data before Google officially declared which schema.org types affect rich results.

How to measure whether the file moves the needle at all

Since this topic is GEO, and GEO without measurement is just opinion, the way forward is to instrument before celebrating:

• Segment server logs by AI crawler user-agent (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot) and compare the volume of hits on /llms.txt before and after publishing it.
• In Google Analytics 4, isolate referral traffic coming from domains such as chatgpt.com, claude.ai, and perplexity.ai, observing whether the volume changes over a 60-to-90-day window after publication.
• Check via grep on the access log whether the file actually receives requests, not just whether it exists on the server.
• Treat the experiment as a before/after with a control period, since publishing the file is too low-cost to justify a more expensive controlled test.

If, after that period, none of these signals moves, llms.txt is not being consumed by whoever the site wanted to reach, at least not yet.

When publishing is worth the effort, and when it's just guesswork

For technical documentation sites and developer tools, the calculation is more favorable: coding agents, like the one running behind the PagePilot extension for VS Code, or any client using the server-llm-txt MCP server, actually query llms.txt by design. This audience is verifiable and documented in the specification itself, unlike traffic from mass-market conversational search engines.

For marketing and institutional content sites hoping to be cited in a ChatGPT response, the bet is weaker. The cost of publishing is low, especially if the platform already generates the file, like Mintlify, GitBook, or Wix, but return expectations need to stay modest until OpenAI, Anthropic, or Google publicly confirm that their own search and chat products consume the file. Publishing doesn't hurt; promising a client that llms.txt will generate AI traffic is selling a result that the specification, read carefully, still doesn't guarantee.