What Is llms.txt? How the AI Site Guide File Works
llms.txt: a plain Markdown file, usually published at
/llms.txton a website, that gives AI agents and large language models a short summary of the site and a curated list of links to the pages they should read first.
What is llms.txt?
If you ask what is llms.txt, the shortest useful answer is that it is a guidebook for machines. A human visitor can scan a homepage, ignore the navigation and find the right page. An AI agent fetching a website has a harder time: HTML pages wrap their content in menus, scripts, cookie banners and ads, and converting all of that back into clean text is slow and imprecise. An llms.txt file skips that step by handing the agent a compact, human-written overview with links to the content that matters.
The idea was proposed by Jeremy Howard, co-founder of the research lab Answer.AI, in a 2024 post titled /llms.txt: a proposal to provide information to help LLMs use websites. That post introduced the format. The specification lives at llmstxt.org, which describes it as “a proposal to standardise on using an /llms.txt file to provide information to help agents use a website.”
Two points are worth stating up front. First, llms.txt is a proposal, not a formal web standard from a body such as the IETF or W3C. Second, it is mainly about helping an AI tool answer a question in the moment (what the spec calls inference), not about controlling whether a model trains on a site.
Why llms.txt matters
Software buyers increasingly ask AI assistants to explain a product, compare vendors or walk them through setup. Coding agents fetch documentation to get an API call right. In both cases the agent has a limited context window, the amount of text it can hold at once, and every wasted token costs time and money. The original proposal puts the problem plainly: context windows are too small for most websites in their entirety, so site owners are better placed than any crawler to say which pages matter.
That is the core argument for the file. Site authors know which pages explain the product best, which reference pages are authoritative and which are secondary. An llms.txt file lets them say so directly instead of leaving an agent to guess from a sitemap with thousands of URLs.
For a software company, the practical upside is control over the starting point. If an AI agent reads a vendor’s docs or pricing explanations through a curated file, the summary it builds is more likely to reflect what the vendor actually wants a buyer to know. That connects llms.txt to the wider work of generative engine optimization, shaping content so AI systems use and cite it, though the file on its own is a small piece of that work.
How llms.txt works
The file is ordinary Markdown, which keeps it readable by people and models alike. The specification also defines a precise order of sections, so simple programs can parse it with standard tools.
The required and optional sections
According to the llmstxt.org specification, a file contains these parts, in this order:
- An H1 with the name of the project or site. This is the only required section.
- A blockquote with a short summary containing the key information needed to understand the rest of the file.
- Zero or more paragraphs or lists (anything except headings) with more detail about the project and how to use the linked files.
- Zero or more H2 sections of “file lists”. Each list item is a Markdown link, optionally followed by a colon and a note about the file.
A section named ## Optional has a special meaning by convention: it holds secondary links an agent can skip when it needs a shorter context.
A minimal example
A simple file for a fictional sales training site might look like this:
# Example Sales Academy
> Example Sales Academy publishes free courses on discovery calls,
> objection handling and pipeline reviews for B2B software sellers.
All course pages have a Markdown version at the same URL with .md added.
## Courses
- [Discovery calls](https://example.com/courses/discovery.md): Question frameworks and call structure
- [Objection handling](https://example.com/courses/objections.md): Common objections and responses
## Optional
- [Course archive](https://example.com/archive.md): Older material kept for reference
The spec suggests linking to clean Markdown versions of pages where possible, served at the same URL with .md added, so an agent can read them without parsing HTML. The current version of the proposal also recommends standard link relations: rel="alternate" type="text/markdown" to point a page at its Markdown version, and rel="describedby" to point it at the llms.txt file that covers it. A file can sit at the root or at a subpath, so /docs/llms.txt covers everything under /docs/.
What llms-full.txt is
The base file is deliberately short: it is a map, and the detail lives behind the links. Some sites also publish a companion file that bundles the full content into one large document, most often named llms-full.txt. Svelte, for example, links a complete documentation file (svelte.dev/llms-full.txt) from its llms.txt alongside shorter abridged and compressed versions.
llms-full.txt is a widely used convention rather than part of the core spec. The original Answer.AI post described a related approach for the FastHTML project: expanded files named llms-ctx.txt (without the optional links) and llms-ctx-full.txt (with them), generated from the base file. A full file saves an agent many separate fetches, but it can be far too large for a single context window, which is the exact problem the short file was designed to avoid.
Examples of sites that publish llms.txt
The format is used most heavily for software documentation, where coding agents follow it to find API references and tutorials. A few live examples:
- Anthropic. The Claude developer platform llms.txt (
platform.claude.com/llms.txt) opens with an H1 and an overview of the API documentation and developer resources. - Cloudflare. The Cloudflare developer docs llms.txt (
developers.cloudflare.com/llms.txt) works as an index of indexes: each product links to its own llms.txt with a full list of that product’s pages. - Svelte. The Svelte llms.txt (
svelte.dev/llms.txt) offers several documentation sets of different sizes, from compressed to complete, so an agent can choose based on how much context it has. - Vercel. The Vercel llms.txt (
vercel.com/llms.txt) gives a one-line summary in a blockquote, then points to machine-readable documentation and platform resources.
According to the specification’s author, AI labs including OpenAI, Anthropic and Google publish llms.txt files for their own developer docs, and documentation platforms such as Mintlify and GitBook, plus WordPress plugins such as Yoast SEO, can generate one automatically. Inside SaaS Sales publishes its own llms.txt too, pointing agents to the Library, Reviews and Sales Templates sections.
llms.txt vs robots.txt vs sitemap.xml
The name deliberately follows /robots.txt and /sitemap.xml, two long-standing files at a site’s root. The three do different jobs.
| File | Purpose | Format | Main audience |
|---|---|---|---|
| llms.txt | Curated summary and links to the most useful content | Markdown | AI agents and language models answering a question |
| robots.txt | Rules for which parts of a site automated crawlers may access | Plain text directives | Search engine and other crawlers |
| sitemap.xml | A list of URLs available for crawling, with optional metadata | XML | Search engines |
robots.txt is about permission. The Robots Exclusion Protocol (RFC 9309) defines rules that crawlers are requested to honor when accessing a site. It says what a crawler should not fetch; it says nothing about what a page means. An llms.txt file does not grant or block access, so a site that wants to keep certain AI crawlers out still needs robots.txt for that.
sitemap.xml is about completeness. The sitemaps protocol lets a site list the URLs it wants search engines to know about. The llms.txt spec explains why a sitemap is not a substitute: it often will not list Markdown versions of pages, it does not include helpful external links, and it usually covers far more content than fits in a model’s context window.
llms.txt is about curation. It is short by design, written for a reader that is about to use the content to answer a question, and it can point outside the site when an external reference explains something better.
Adoption and what Google has said
The honest picture is mixed. On the publishing side, the format has spread quickly through developer documentation. The spec’s author reports that thousands of sites publish an llms.txt file and that Chrome’s Lighthouse audits sites for one as part of its agentic browsing checks.
On the search side, Google has been direct. Its guide to AI features and your website, which covers AI Overviews and AI Mode, states that site owners do not need to create any extra machine-readable files, AI text files or markup to appear in these features. In other words, Google does not require llms.txt for its AI search results, and the same guide points site owners back to standard SEO practice instead.
That leaves llms.txt in a specific niche. It is most useful where an agent actively fetches a site to help someone, such as a coding assistant reading docs or a chat assistant researching a product. It is not a ranking lever for Google, and no major AI platform publicly promises to cite a site because it has one. Treat it as low-cost housekeeping that can help agents, not as a shortcut into AI answers. The broader work of AIO and answer engine optimization still depends on clear, useful, crawlable pages.
How to create an llms.txt file
- Pick the scope. Decide whether the file covers the whole site at
/llms.txtor a section such as/docs/llms.txt. - Write the H1 and summary. Name the site, then write a one to three sentence blockquote that explains what it is and who it serves.
- Group the key links under H2 headings. Use plain section names such as Docs, Guides or Pricing, and add a short note after each link.
- Move secondary material to Optional. Archives, changelogs and long references belong there.
- Offer Markdown versions where you can. Clean
.mdpages are easier for agents to read than full HTML. - Test it. The spec suggests asking an agent questions about your content while giving it only the llms.txt file as a starting point.
- Keep it current. Many docs platforms and plugins regenerate it automatically.
Related terms
- What is AIO?: the broad practice of making content easy for AI answer engines to find, understand and cite.
- What is generative engine optimization?: shaping content so AI systems that write answers use it as a source.
- What is answer engine optimization?: the practice of becoming the direct answer to a question.
- AI SDR: AI agents that run top-of-funnel prospecting and often research buyers through the same AI tools.
- Browse the full Library or see the best AI sales tools.
FAQ
Is llms.txt an official web standard?
No. It is an open proposal published at llmstxt.org, maintained in a public GitHub repository with community input. It is not a standard from the IETF or W3C, unlike robots.txt, which is defined in RFC 9309.
Does llms.txt help a site rank in Google AI Overviews?
Google says it does not need one. Its AI features guide states that site owners do not need to create extra machine-readable files or AI text files to appear in AI Overviews or AI Mode. Standard SEO practices still apply.
What is the difference between llms.txt and llms-full.txt?
llms.txt is a short map: a summary plus curated links. llms-full.txt is a common companion convention that bundles the full text of the linked content into one large file, so an agent can read everything in a single fetch. Only the short file is defined in the core spec.
Does llms.txt block AI crawlers or AI training?
No. llms.txt does not control access. To ask crawlers, including AI crawlers, not to fetch parts of a site, use robots.txt. The spec describes llms.txt as information used on demand when an agent is helping a user.
Where should the llms.txt file go?
At the site root, at /llms.txt, for a whole-site guide. The spec also allows a file at any subpath, such as /docs/llms.txt, which covers only the pages under that path. Where more than one file applies, agents should use the most specific one.
Sources
Checked November 15, 2026.