llms.txt: what the spec really requires before you create the file

Brazilian brands have started asking whether they need to publish an llms.txt file before it becomes the de facto standard for GEO (generative engine optimization). The question makes sense: the official specification, written by Jeremy Howard in September 2024, has just received a version 2 (last modified on August 10, 2026) and now even shows up in Chrome's Lighthouse audits, which check for the file's presence as part of agentic browsing tests. Before rushing to create the file, it's worth understanding what the spec actually requires today, what it is not, and where there still isn't measurable return.

What the file is (and what it is not)

llms.txt is a Markdown file designed to be read by language models and agents. The logic behind it: HTML pages are made for humans. They wrap information in navigation, ads, and JavaScript, and converting that back into clean text is expensive and imprecise. Since the context window is still finite and every wasted token costs time and money, the proposal offers agents a curated, concise entry point in a single accessible location.

The detail the source is careful to reinforce: the file doesn't replace anything that already exists. It coexists with robots.txt and sitemap.xml, but serves a different function:

| File | For whom | When it's used | |---|---|---| | robots.txt | Indexing bots | Defines what can be accessed | | sitemap.xml | Traditional search engines | Lists all indexable pages | | llms.txt | AI agents / LLMs | On demand, when the agent needs to understand a topic |

Jeremy Howard is explicit about the intended use: llms.txt is meant for inference, not for training. In other words, the value shows up when an assistant answers a user's question and looks for context about your product, not in some future training cycle (although training runs may also make use of the information).

What the spec actually requires

Here's the point that dissolves much of the brands' anxiety: the required structure is minimal. The specification lists the sections in this order, and only one of them is mandatory:

• An optional BOM (byte-order mark);
• An H1 with the project or site name (the only mandatory section);
• A blockquote with a short project summary;
• Zero or more Markdown text blocks (paragraphs, lists) without headings;
• Zero or more sections delimited by H2, containing lists of links in the format [name](url): optional notes.

In practice, a valid llms.txt can be as lean as a single title. The full format looks like this:

# Title

> Optional description goes here

Optional details go here

## Section name

- [Link title](https://link_url): optional link details

## Optional

- [Link title](https://link_url)

The section called ## Optional has a semantic function: by convention, it holds secondary information, the links the agent can skip when it needs shorter context. It's an explicit priority signal, not just an extra list.

Another important detail: the file doesn't have to sit only at the root. It can live at any subpath, and a /docs/llms.txt covers everything under /docs/. When more than one file applies, the agent should use the most specific one. It was precisely this path flexibility that made the spec favor the llms.txt pattern over the /.well-known/ path from RFC 8615: someone publishing on GitHub Pages, for example, controls a directory but never the root of the shared host.

The part almost no one implements: the .md files

This is where the real work lives, and where a lot of brands stall halfway through. llms.txt on its own is just an index. The full proposal asks that pages with information useful to agents also have a clean Markdown version at the same URL, either with .md appended (page.html.md) or with the extension swapped (page.md). The links inside llms.txt should point to that LLM-friendly content, not to HTML full of noise.

For clients to find these files, the spec recommends standard link relations: rel="alternate" type="text/markdown" points to the page's Markdown version, and rel="describedby" points to the llms.txt that covers it. This can be done via a element in the HTML or as an HTTP header, which lets you configure everything at the CDN level without touching the pages:

Link: </docs/page.html.md>; rel="alternate"; type="text/markdown", </docs/llms.txt>; rel="describedby"

The FastHTML project, also by Howard, follows both proposals: it keeps an llms.txt in /docs/ and serves every documentation page at a URL with the .md extension. All nbdev projects now generate .md versions of every page by default.

Who's already adopting it, and why documentation leads

Version 2 didn't emerge from theory. According to the source, thousands of sites publish the file, documentation platforms generate it automatically, and the AI labs themselves publish their own: OpenAI, Anthropic, and Gemini have llms.txt files for their respective developer docs.

The heaviest use is in software documentation, where coding agents follow the file to find API references and tutorials when getting a call right. That's no coincidence: it's the scenario where the agent truly needs precise context and the error is immediately visible (the code doesn't run). The same structure serves any site that wants to offer a guided path, from a company's policies to a personal resume, but the clear return today sits in the technical niche.

Automatic generation is already mature across several platforms, which lowers the entry cost:

• Mintlify and GitBook: generate llms.txt (and, in Mintlify's case, Markdown versions of each page) for the sites they host;
• Yoast SEO and AIOSEO: WordPress plugins that create and maintain the file;
• Wix: generates an llms.txt for every site;
• Docusaurus and VitePress plugins, plus a Drupal Recipe for Drupal 10.3+, cover documentation stacks.

When it still isn't worth it, and how to measure

The point that separates engineering from hype: llms.txt is a proposal open to community input, hosted in a GitHub repository with public discussion, not a standard ratified by any standards body. Chrome auditing its presence in Lighthouse is de facto adoption, not an obligation. Nothing breaks if you don't have the file, and no search engine penalizes you for it today.

For a Brazilian brand deciding on this, the sensible approach is to look at the type of content:

• It's worth prioritizing when you have technical documentation, a public API, or a knowledge base that coding agents and assistants consult frequently. If the platform (Mintlify, GitBook, WordPress with Yoast) generates the file on its own, the cost is close to zero and there's no reason not to turn it on.
• It's not worth forcing a handcrafted llms.txt, with dozens of manually maintained .md versions, on an institutional marketing site with no content an agent would need to fetch. The effort of keeping the Markdown files in sync with the HTML turns into technical debt with no audience on the other end.

The spec itself offers the most honest validation method: test your file by asking an agent questions, giving it only your llms.txt as a starting point. If the model answers correctly about your content using just that index, the file is doing its job; if it gets lost or hallucinates, the problem lies in the curation of the links and descriptions, not in the absence of the file.

This is how you measure before and after here: not by SERP position, which llms.txt doesn't affect, but by the quality of the answer an assistant gives about your brand. Directories like llmstxt.site, directory.llmstxt.cloud, and llmstxthub.com list already-published files and serve as a reference for calibrating your own. Traffic has changed, and it's worth keeping up with the spec, but the decision remains an engineering one: publish where there's an agent on the other end to consume it, and measure the response before declaring victory.