llms.txt and robots.txt: two files, two jobs.
robots.txt tells a crawler where it may go; llms.txt tells a model what a site is and where to find what matters.
Both files live at the root of a site, both are plain text, and their names look alike — which is why they get confused so often. But their jobs are entirely different, and neither replaces the other. Understanding the difference means expecting the right thing from each.
robots.txt is a thirty-year-old standard, recognized by every search engine, and it is an ACCESS protocol: which path may be crawled, which may not. llms.txt is a new proposal, officially supported by no engine, and it is a MAP: a summary of the site, its main sections, and a list of its important pages.
What each file does
- robots.txt — access permission. Which agent may enter where. Widely recognized and respected.
- robots.txt — sitemap declaration. The starting point of discovery.
- llms.txt — a one-paragraph summary of what the site is.
- llms.txt — an annotated list of the main sections and important pages.
- llms.txt — verifiable facts, gathered in one tidy place.
One thing needs saying plainly: there is no evidence that Google reads llms.txt, and the company has announced no such support. The picture is similar for other answer engines — some may read it, some will not. Publishing this file is not a guarantee; it is a low-cost bet.
So why write it anyway? There are three reasons, and all three are independent of the file itself.
First, the exercise of writing it clarifies the site itself. Being forced to summarize a site in a single paragraph and to describe its main pages in one sentence each makes you see what the site actually says. That clarity later carries over into page titles and descriptions too.
Second, the cost is close to zero. A file that comes out of the build pipeline updates itself as pages change. If it is not an item that needs manual upkeep, even a low-probability return is reasonable.
Third, even if the standard never takes hold, the content itself is useful elsewhere: the same summary and the same list of facts can serve the homepage description, the schema markup, and the social cards.
llms.txt is not a guarantee but a low-cost bet — and its real benefit appears while you are writing it.
The implementation on this site
This site’s llms.txt comes out of the build pipeline and is never edited by hand. It contains the site’s definition, the declaration of its three languages, an annotated list of the main pages, the case studies, the journal articles, and a section of “verifiable facts.”
That last section is a deliberate choice: the file carries facts, not marketing language. Who the founder is, how many languages, which technology, which projects. The kind of information a model can verify and cross-check against other sources.
And one consistency rule: every URL in the file must be real, and every claim must be verifiable on the site itself. The verification suite measures, on every run, that the links in llms.txt match the blocks on screen — if the file ever starts lying, the test remains.
Which one comes first
The order is not up for debate: robots.txt comes first. Without access permission, no map is of any use. When a site is being prepared for answer engines, the first place to look is robots.txt and the second is the server logs — is the crawler actually coming?
llms.txt can be left for last. If access works, the content structure is clean, and the schema is in place, this file is a small extra layer. If access is broken, the file means nothing.
The practical summary: robots.txt is an obligation, llms.txt is a choice. Get the first one wrong and your visibility shuts down; skip the second entirely and you lose nothing — but the writing exercise itself does the site’s language good.
What goes into the file
The format of llms.txt is defined as a proposal and is not strict, but there is a structure that works. A title, a summary paragraph, annotated link lists under section headings, and — where they exist — verifiable facts.
- Title — the site’s name. Short, written as a brand.
- Summary — what the site is, in a single paragraph. The language of definition, not of marketing.
- Sections — the main page groups, with a one-sentence note next to every link.
- Facts — verifiable information: founding year, languages, technology, measured numbers.
- Links — the sitemap and, if present, the feed addresses.
The annotation sentences are the most valuable part, because they let a model understand a page without opening it. A sentence’s job is to say which question that page answers — not to repeat its title.
And having the file produced by the build pipeline is always better than writing it by hand. A hand-written map goes stale as pages change, and a stale map is worse than no map at all — it points a model at pages that do not exist.
Consistency: a file that does not lie
The one real requirement of a map is that the place it points to is actually there. When a page listed in llms.txt is unpublished or its address changes, the file quietly starts lying — and a map that sends a model to pages that do not exist is worse than no map at all, because it breaks trust. That is why generating the file from the build, rather than by hand, is a necessity and not a preference: as the pages change, the map changes with them, and no human error lives in between.
The same consistency applies to claims. The file’s “verifiable facts” section should carry information that also appears, and can be measured, elsewhere on the site. How many languages, which technology, which projects — all of it should be verifiable on the pages too. On this site, the verification suite measures on every run that the link list in llms.txt matches the blocks on screen; if the file ever drifts, the test turns red and the difference is visible at deploy time.
One last practical note: the file’s content type must be plain text, and it should be readable when opened in a browser. Some server configurations serve files with unknown extensions as downloads, or send them with the wrong type; in that case the file technically exists but cannot be read. The check is simple — open the address in a browser, make sure it renders as text, and confirm that the response code is 200. These three checks can and should be automated, because a server configuration can quietly change one day.