Does llms.txt work? What the 2026 data says.

There is an llms.txt at the root of this site and it has its own scene on the home page. So the question is not left for someone else to ask: what the data says is written here.

The answer has two parts and both are true. As a lever for citations the file does not work — three separate studies measure that. As an interface for agents it does work, and the tools using it are known. The two claims do not refute each other, because they are about different jobs.

The measured side: not a lever for citations

In May 2026 Ahrefs scanned the server logs of the 137,210 domains receiving traffic in its web analytics. The result in one sentence: 97 per cent of llms.txt files received no request at all that month. Of roughly 38,000 valid files, only about 1,100 were read.

The breakdown by bot is sharper still. The largest group requesting the file is SEO auditing tools — 21.7 per cent of all requests. AI training crawlers stay at 5.3 per cent (GPTBot 4.51, ClaudeBot 0.80, DeepseekBot 0.02) and the retrieval bots that feed citations at 1.1 per cent. What reads the file, in other words, is mostly the tool auditing it rather than the model using it.

SE Ranking’s scan of roughly 300,000 domains put adoption at 10.13 per cent — one site in ten. The study’s second finding matters more: once site authority, schema density and content recency are controlled for, no measurable citation lift attributable to llms.txt remains. Among the largest sites by traffic adoption is close to absent; different samples give different numbers, but the tendency points the same way.

Google’s side is plain too. The AI optimisation guidance of 15 May 2026 says directly that llms.txt is not needed for AI Overviews or AI Mode. Gary Illyes said that ranking in generative surfaces takes “normal SEO” and noted the file would not be crawled; John Mueller compared it to the discredited keywords meta tag.

The unmeasured side: the file is not dead

In the same period there is a group genuinely using it. Anthropic publishes its own documentation in this form. Perplexity says it reads llms.txt when deciding which page to fetch. The agents inside code editors — Cursor, Continue, Cline — use the file directly. What these have in common is that none of them is a search engine.

Chrome Lighthouse 13.3, released in May 2026, added an “Agentic Browsing” category that looks at four things: llms.txt, WebMCP, the page’s accessibility tree and layout shift. The interesting part is that the category does not score — it produces no weighted number, because the standards of the agentic web have not settled and the aim for now is to gather data.

That audit’s own measurement is instructive as well: of 186 sites serving a file, only 64 had a valid one. The remaining 122 had a problem — no H1 heading, content too short, or no links at all. Two thirds of those who placed the file, then, did not get the format right.

The right frame: an interface, not a lever

The two sets of findings do not conflict. Placing llms.txt to be cited more often by a search engine does not work; the data measuring that is clear. Placing it to introduce the site to an agent in a single request does work; the tools doing so are known. The confusion comes from giving both the same name.

The distinction has a practical counterpart. If paid crawling models spread, fetching a curated index plus three target pages becomes cheaper than crawling fifty — and at that point curation stops being a visibility tactic and becomes a cost decision.

What we did on this site

This site’s file had swollen to 30.8 kilobytes. Measured: 85 per cent of the bulk was two sections — the guide list at 18.1 and the glossary list at 8.2 kilobytes. A file meant to be an index had turned into a directory, and that is precisely what the spec does not ask for.

The file was split in two: a 4.5-kilobyte index at the root and a full directory linked from it. The index only orients; the full directory is for deep reading and is marked not to be indexed. The pattern is not an invention — Anthropic, Vercel and LangGraph use the same pair, and it is the most common setup in 2026.

Two format errors closed in the same round: the section headings now really carry a list of links, and there is a single summary block in the file. Both are what the Lighthouse audit above looks at — and where it eliminated those 122 sites.

Why we still publish it

Because the file costs us one line in the generator. It is not maintained by hand; it is produced from the collections and grows on its own when a new page appears. No risk of going stale, no maintenance burden.

And because we put the expectation in the right place. This file brings no rankings, no citations, promises no visibility. It gives a reading agent a tidy map — that is all. That something does a small job well is no reason not to publish it.

We read the studies saying llms.txt does not work. We publish it anyway — but now we know why.