llms.txt: the file that guides AI agents on a website.

What is llms.txt?

llms.txt is a Markdown file placed at a domain's root (https://example.com/llms.txt) that gives large language models a short, curated map of a site's most useful pages and resources. It was proposed on September 3, 2024 by Jeremy Howard, co-founder of Answer.AI and fast.ai, and its specification lives at llmstxt.org. Established

The original argument is concrete: model context windows are narrow relative to a typical website, and converting HTML pages (navigation, scripts, ads) into clean LLM content is difficult and imprecise. Rather than letting a model crawl an entire site for every question, the author hands it a readable list of the pages that matter. The choice of Markdown, rather than a structured format like XML, is deliberate: these files are meant first to be read by models and agents. The video below gives an accessible overview.

How it is structured

The specification defines a minimal grammar, parsable in a few lines of code: a single H1 heading (the project name), an optional blockquote summary, then H2 sections containing lists of links, each item in the form - [name](url): note. Only the H1 is strictly required.

llms.txt, robots.txt, sitemap.xml: three files, three roles

The most common confusion treats llms.txt as a replacement for the sitemap or robots.txt. These are three complementary root conventions answering different questions. Established

Does it work? The reality in 2026

This is where epistemic honesty matters. llms.txt is carried by strong enthusiasm, but the facts call for nuance. On adoption, several documentation platforms publish the file: Anthropic, Cloudflare, Stripe, Mintlify, Vercel, Perplexity. Publishing a file is not the same as confirming you consume it on the receiving end. Emerging

On the engine side, Google's position is explicit and repeated. In June 2025, John Mueller wrote: "FWIW no AI system currently uses llms.txt", adding that server logs show it: chatbots fetch the pages, not the llms.txt file. Gary Illyes confirmed Google does not crawl it and has no plans to, and Google's official AI optimization guide recommends skipping it for Search. Established

In 2026, Mueller sharpened the underlying argument on the Search Off the Record podcast: a self-reported file cannot help an LLM differentiate sites from one another, since each can claim to be the best. For discovery, he points back to HTML pages and internal links. He carves out one clear exception: "if an agent is already on your website, some automated system can be helpful". The distinction is valuable: llms.txt is more about navigation (a directory for the agent already inside) than discovery (choosing which site to visit). Google itself promotes WebMCP for agentic interaction. Emerging

Where llms.txt has value today

The sober reading is not a rejection. llms.txt has real use cases, as long as we name them for what they are.

• Grounding at inference. When a user submits your domain to Claude, ChatGPT, Perplexity or a homegrown RAG pipeline, several of these tools fetch /llms.txt first. A curated file gives them the pages to ground the answer on, instead of whatever the open web surfaces.
• Developer tools. Cursor, Windsurf and several MCP integrations explicitly fetch the file to load up-to-date documentation as context.
• Documentation and API sites. The highest ROI, especially paired with /llms-full.txt: a clean channel for developers and their assistants.
• The internal benefit. Writing the file forces a team to agree on the dozen pages that truly represent the project. That curation exercise has value on its own.

llms.txt within the UCP framework

Placing llms.txt inside the four-layer framework avoids a category error. It is a brick of the discovery layer, alongside the sitemap, robots.txt and agent surfaces like agent-card.json. It belongs to navigation: helping an agent already present find its way.

It touches neither the semantic layer (making catalog data machine-addressable, via schema.org), nor the transaction layer (making the offer purchasable by an agent, via UCP or WebMCP). That is the limit the current debate points to: a discovery file says where to look, it does not make the offer retrievable or transactable. For catalog readability and citation, it is the GEO readiness work that matters; for delegated purchase, it is implementing UCP.

Creating and validating your llms.txt

Creation is deliberately trivial: a text editor is enough. The smallest valid file is an H1 followed by one section. For most stacks (Astro, Next.js, Cloudflare Pages), just drop the file in the public/ directory; it is served as-is at /llms.txt as text/plain. The difficulty is not technical, it is editorial: choosing the right pages and keeping the URLs current.

Frequently asked questions