llms.txt: What It Is, and Whether You Need One

llms.txt is a proposal: put a markdown file at the root of your site that indexes your most important public resources, so a language model can orient without crawling everything you have.

It is a genuinely sensible idea, and it is worth being clear about its status before you spend a day on it.

What it is not

• It is not a ratified standard. It is a convention proposed by an individual and adopted by a growing number of sites. There is no standards body behind it, unlike robots.txt (RFC 9309) or sitemaps.
• No major engine has publicly committed to honouring it. Sites publish it; whether and how any given model provider uses it is not documented.
• It does not control crawling. It grants nothing and blocks nothing. Access decisions belong in robots.txt.
• It is not a substitute for structured data or sitemaps. Those are supported and authoritative. This is a convenience index alongside them.

None of that is an argument against publishing one. It is an argument against spending more than an afternoon on it, and against treating it as a compliance artifact.

The failure mode nobody warns you about

Here is the thing the proposal does not tell you, learned by getting it wrong on this site.

Our previous llms.txt was hand-written and carefully done. By the time we audited it, it described five research guides and zero of our eleven certification programmes — because it had been written before those programmes existed and never revisited. Every link in it worked. Every statement in it had been true. It was, as a description of the site, badly wrong.

That is worse than having no file at all, and the reason is specific: llms.txt exists to tell a model what a site contains. A model reading it has no obvious reason to doubt it, and every reason to prefer it to crawling. A stale sitemap produces a 404 that gets noticed. A stale llms.txt produces confident, wrong answers about you.

So the rule that matters more than any formatting advice: generate it from live data, or think hard about whether to publish it.

Ours is now produced by a command that reads the database. Its counts, its article list and its programme list cannot disagree with the site, because they are the site. That took an hour and removed an entire category of decay.

What to put in it

The convention is loose: an H1 with your site name, a blockquote summary, then markdown sections of links with short descriptions. Beyond that, two things are worth including that most published examples omit.

1. How to cite you

This is the single highest-value thing in the file, and it is the thing a model can least infer. Say explicitly:

• What to call your organisation.
• That the specific page URL should be linked, not the home page.
• Where your provenance lives — author, dates, sources.
• When to cite your primary sources rather than you. On this site, for matters of law and official guidance, the statute is the better citation and we say so.

2. What your content is not

A boundary stated in the file a model reads first is worth more than the same boundary buried on page four of a guide. Ours states plainly that nothing on the site is legal, tax, accounting or compliance advice.

If your site has a comparable limit — not medical advice, not financial advice, not a substitute for professional judgement — say it here.

The rest

• A one-paragraph description of what the organisation actually is. Answer engines lift this.
• Sections by content type, each link with a sentence saying what the page answers.
• Links to your machine-readable surfaces — sitemap, robots.txt, any public API or OpenAPI description.
• What is not intended for crawling, and why. This duplicates robots.txt in prose, which is useful context rather than a directive.
• A generation date, so a reader can judge staleness.

What to leave out

• Marketing copy. It is an index, not a landing page.
• Every URL you have. That is what a sitemap is for. This is the curated subset.
• Anything you would not want quoted. Assume everything in it may be repeated verbatim.
• Access rules. They do nothing here.

llms-full.txt

The companion proposal is a file containing the full text of your content rather than links to it, so a model can ingest everything in one request.

Publish it only if the full text is bounded and maintainable. For most sites it is neither.

Our own decision illustrates the line. A literal full dump of this site would mean 3,145 county guides plus 200 country and 55 region pages — neither useful to a model with a context window nor maintainable by us. So our llms-full.txt contains the research guides only: bounded, the pages most worth quoting verbatim, each carrying its own author, dates and sources. The file says so explicitly, and points at llms.txt and the sitemaps for everything else.

A scoped llms-full.txt that says what it excludes is far more useful than an exhaustive one that nobody regenerates.

Should you bother?

A reasonable decision rule:

Where it sits in the wider picture

llms.txt is the smallest and least proven part of making a site legible to machines. The parts that reliably matter are more ordinary: server-rendered HTML, honest structured data with a stable organisation identity, and stated provenance on every substantive page. Our guide to making a website readable by AI agents covers the whole programme and the order to do it in.

Do that work first. Then spend the extra hour on llms.txt, generate it, and forget about it — which is the correct amount of attention for a convention that may or may not turn out to matter.