llms.txt is a young, still-informal convention — a plain-Markdown file at your site root meant to give AI systems a curated summary of what your site is and where the important pages live, instead of making them guess from a crawl. Adoption is real but uneven, and support isn’t guaranteed by any major AI product. That’s worth saying plainly before the checklist, not after it.
How uneven? In our October 2026 scan of the top 10,000 domains, 14.4% of the sites that answered served a valid llms.txt (18.5% among the top 1,000), and another 409 returned an ordinary web page at /llms.txt instead of a file.
What actually helps
- A one-paragraph summary of what the site does, written the way you’d explain it to a person who’s never heard of you — not marketing copy, a plain description.
- Links to your most important pages, grouped and labeled, so a system reading the file gets a map instead of a link dump.
- A note on what’s out of scope, if relevant — telling a system what you *don’t* cover is sometimes as useful as telling it what you do.
- Keeping it current. A stale llms.txt pointing at a redesigned site is worse than no file, the same way a stale sitemap is worse than none.
What only looks useful
- Keyword-stuffed summaries. The file is meant to be read by a language model, not ranked by a classic search index — stuffing it with target phrases doesn’t help and reads as exactly the kind of manipulation the spec’s authors were trying to avoid inviting.
- Duplicating your entire sitemap. llms.txt is meant to curate, not enumerate. A file that lists every URL on your site provides no more signal than the sitemap already does, and misses the point of the format.
- Claims you wouldn’t put in your own marketing copy. If it wouldn’t survive scrutiny on your homepage, it doesn’t become true by being on a file aimed at AI crawlers instead of people.
- A file with no maintenance owner. We’ve seen llms.txt files that shipped once and were never revisited — worse than not having one, because it actively tells a system something that’s no longer accurate.
Where this fits with robots.txt
llms.txt and robots.txt do different jobs and shouldn’t be confused: robots.txt is a permission file (who’s allowed to crawl what), llms.txt is a curation file (here’s what matters, for the crawlers you’ve already decided to allow). Our own AI-crawler checklist covers the permission side, and our robots.txt generator writes the file; this post covers the curation side. Getting one right without the other is only half the job.
