llms.txt

What Is llms.txt?

llms.txt is a proposed web standard: a markdown file at a site's root path (/llms.txt) that gives large language models a curated map of the site's most important content, with links to clean, markdown-friendly versions of key pages. It exists because HTML pages full of navigation and scripts waste an AI system's limited context.

Key Takeaways

• The file is written in markdown, not a directive syntax. It contains an H1 with the site name, a short summary, and sections of annotated links to the content that matters most.
• It was proposed by Jeremy Howard of Answer.AI in September 2024, and adoption concentrated in developer tool and documentation sites, where AI-assisted usage is heaviest.
• llms.txt is an offering, while robots.txt is a restriction. One says "here is our best content in a convenient form", the other says "here is what you may not crawl".
• A companion convention, llms-full.txt, inlines the full documentation text in one file so an agent can ingest everything in a single fetch.
• As of 2026 no major AI provider has committed to honoring the file in training or live retrieval, so treat it as low-cost preparation with uncertain reach rather than a proven acquisition channel.

How It Works

The proposal standardizes both location and shape. The file lives at the web root, and its structure is deliberately parseable: an H1 title, a blockquote summary of the site, optional free-form notes, then H2-headed sections containing markdown link lists, where each link may carry a one-line annotation explaining what the page covers. An optional final section named "Optional" marks content that can be skipped when an agent is short on room in its context window. Because the format is plain markdown, both humans and programs read it without special tooling.

The second half of the convention addresses the pages themselves. Sites are encouraged to serve markdown twins of important pages, reachable by appending .md to the URL, so a model can fetch documentation without wading through headers, cookie banners, and JavaScript. Documentation platforms made this nearly free: Mintlify, and frameworks and hosts in its wake, generate llms.txt and markdown variants automatically, which is why thousands of developer-facing sites shipped the file within a year of the proposal. Originality.ai's tracking study across more than 3 million monitored websites found adoption grew 8.8x in twelve months, from 4,088 sites in June 2025 to 36,120 by May 2026 [1].

Consumption is where expectations need care. The realistic consumers today are agentic tools that fetch pages on demand: a coding agent told to integrate an SDK can pull the vendor's llms.txt, follow the links, and load exactly the relevant docs. Whether the major AI search and chat products consult the file is undocumented and widely doubted, and server-log studies show little evidence of systematic llms.txt fetching by their crawlers. An Ahrefs log study put numbers on the gap: 28% of 137,210 domains using its web analytics published an llms.txt file, yet 97% of those files received zero traffic in May 2026 [2]. Publishing it is cheap insurance, and for teams optimizing their visibility in AI answers that is usually justification enough.

Example

A payments API company keeps hearing that coding agents hallucinate outdated endpoints from old blog posts instead of using current docs. They publish /llms.txt listing their quickstart, authentication guide, API reference, and migration notes, each with a one-line description, and configure their docs platform to serve every page as markdown at the .md twin URL. Now when a customer's coding agent is asked to add payments, it fetches the index, follows two links, and works from the current v3 reference, about four thousand tokens of clean markdown instead of hundreds of thousands of tokens of rendered HTML. Integration bug reports traceable to stale examples drop noticeably the following quarter.

What People Get Wrong

The biggest misconception is that llms.txt controls AI crawlers the way robots.txt controls search crawlers. It grants nothing and forbids nothing. Blocking AI training bots still happens in robots.txt user-agent rules, and nothing obliges any model provider to read llms.txt at all. The file is closer to a sitemap crossed with a reading guide: purely advisory, valuable exactly to the extent a consuming tool chooses to use it. Publishing one while leaving your actual content unparseable, or expecting it to move AI citation share by itself, misreads what it does.

FAQ

Does Google or OpenAI use llms.txt? Neither has announced support, and Google representatives have publicly played down the idea. The dependable consumers are agent tools that fetch documentation live, such as coding assistants, rather than training pipelines or AI search indexes.

What is the difference between llms.txt and llms-full.txt? llms.txt is an index of links with annotations; llms-full.txt concatenates the full text of the documentation into one large markdown file. The full variant suits one-shot ingestion into a large context, while the index suits selective fetching, and large sites often publish both. Counting llms-full.txt and the related ai.txt format alongside llms.txt, Originality.ai's year-long study found 38,980 sites had adopted an LLM-guidance file by May 2026 [3].

Should my site have one? If AI tools plausibly read your content, especially developer documentation, yes: generation is often a checkbox in modern docs platforms and the downside is nil. For a general marketing site the case is weaker, and effort is better spent making the pages themselves clean, fast, and crawlable.

Sources

1. Originality.ai. "LLMs.txt tracking study, twelve-month adoption growth across 3+ million monitored websites." https://originality.ai/blog/llms-txt-tracking-study. Accessed August 2026.
2. Ahrefs. "llms.txt log study, publication and traffic rates across 137,210 domains." https://ahrefs.com/blog/llmstxt-study/. Accessed August 2026.
3. Originality.ai. "LLMs.txt tracking study, combined adoption of llms.txt, llms-full.txt, and ai.txt files." https://originality.ai/blog/llms-txt-tracking-study. Accessed August 2026.