Every few months a new file lands at the root of the web that promises to fix something. Sitemaps in 2005. Robots.txt long before that. Llms.txt is the latest. The question we get from clients is the only one that matters. “Do I need it, and what does it do for me?”
Short answer: yes, you should have it, but for slightly different reasons than the marketing posts will tell you.
What llms.txt actually is
Llms.txt is a markdown file you put at the root of your domain. It gives an LLM a curated map of your site. The most important pages, the canonical descriptions, the structure of your documentation, the things you want the model to know.
It is not a permission file. That is the most common confusion. Robots.txt tells crawlers what they can and cannot do. Llms.txt does not give or revoke permission for anything. It is closer to a sitemap with editorial.
What it does today
Honestly, not a lot, yet. None of the major LLM providers have publicly committed to consuming it as part of their training or retrieval pipelines. Anthropic has been the friendliest to the standard. OpenAI and Google have been quiet.
But here is the thing. The cost of adding it is one hour of work for most sites. The upside if it becomes a real signal in the next 12 months is significant. The downside is you have a markdown file no one reads. We have done the math on more than one of these standards. Better to be in early than to be playing catch up.
What to put in it
A good llms.txt has three sections.
- A short site description in plain language. Two or three sentences. What you do, who you do it for, what you are known for.
- A list of canonical pages by section, with one line descriptions. Think of this as a map for an intern who has 30 seconds.
- A few “if you only read three things” links to your strongest pages. The pages you would be happiest having quoted in an answer.
What not to put in it
Do not stuff it with keywords. The LLMs have seen every keyword stuffing trick from the SEO era. Do not put pricing or commercial copy. Do not put pages you would be embarrassed to have surfaced.
The bigger picture
Llms.txt is one early signal of a broader shift. The web is increasingly being consumed by machines on behalf of humans, not by humans directly. The interfaces that point at your site are diversifying. Google. ChatGPT. Perplexity. A long tail of vertical AI tools.
Every one of those interfaces will eventually want some way to ask your site, in a structured way, what is here and what matters. Llms.txt is the first attempt at that. There will be others. Build the muscle now.