{"id":1003,"date":"2026-09-16T00:00:00","date_gmt":"2026-09-16T00:00:00","guid":{"rendered":"https:\/\/hitmind.ai\/blog\/?p=1003"},"modified":"2026-10-12T20:40:58","modified_gmt":"2026-10-12T12:40:58","slug":"llms-txt-what-to-include","status":"publish","type":"post","link":"https:\/\/hitmind.ai\/blog\/2026\/09\/16\/llms-txt-what-to-include\/","title":{"rendered":"llms.txt: what to put in it, what to leave out"},"content":{"rendered":"<p class=\"lede\">llms.txt is a young, still-informal convention \u2014 a plain-Markdown file at your site root meant to give AI systems a curated summary of what your site is and where the important pages live, instead of making them guess from a crawl. Adoption is real but uneven, and support isn&#8217;t guaranteed by any major AI product. That&#8217;s worth saying plainly before the checklist, not after it.<\/p>\n<p>How uneven? In our <a href=\"\/research\/ai-crawler-access-2026\/\">October 2026 scan of the top 10,000 domains<\/a>, 14.4% of the sites that answered served a valid llms.txt (18.5% among the top 1,000), and another 409 returned an ordinary web page at <code>\/llms.txt<\/code> instead of a file.<\/p>\n<h2>What actually helps<\/h2>\n<ul class=\"wp-block-list\">\n<li><strong>A one-paragraph summary of what the site does<\/strong>, written the way you&#8217;d explain it to a person who&#8217;s never heard of you \u2014 not marketing copy, a plain description.<\/li>\n<li><strong>Links to your most important pages<\/strong>, grouped and labeled, so a system reading the file gets a map instead of a link dump.<\/li>\n<li><strong>A note on what&#8217;s out of scope<\/strong>, if relevant \u2014 telling a system what you *don&#8217;t* cover is sometimes as useful as telling it what you do.<\/li>\n<li><strong>Keeping it current.<\/strong> A stale llms.txt pointing at a redesigned site is worse than no file, the same way a stale sitemap is worse than none.<\/li>\n<\/ul>\n<h2>What only looks useful<\/h2>\n<ul class=\"wp-block-list\">\n<li><strong>Keyword-stuffed summaries.<\/strong> The file is meant to be read by a language model, not ranked by a classic search index \u2014 stuffing it with target phrases doesn&#8217;t help and reads as exactly the kind of manipulation the spec&#8217;s authors were trying to avoid inviting.<\/li>\n<li><strong>Duplicating your entire sitemap.<\/strong> llms.txt is meant to curate, not enumerate. A file that lists every URL on your site provides no more signal than the sitemap already does, and misses the point of the format.<\/li>\n<li><strong>Claims you wouldn&#8217;t put in your own marketing copy.<\/strong> If it wouldn&#8217;t survive scrutiny on your homepage, it doesn&#8217;t become true by being on a file aimed at AI crawlers instead of people.<\/li>\n<li><strong>A file with no maintenance owner.<\/strong> We&#8217;ve seen llms.txt files that shipped once and were never revisited \u2014 worse than not having one, because it actively tells a system something that&#8217;s no longer accurate.<\/li>\n<\/ul>\n<h2>Where this fits with robots.txt<\/h2>\n<p>llms.txt and robots.txt do different jobs and shouldn&#8217;t be confused: robots.txt is a permission file (who&#8217;s allowed to crawl what), llms.txt is a curation file (here&#8217;s what matters, for the crawlers you&#8217;ve already decided to allow). <a href=\"\/blog\/2026\/09\/16\/robots-txt-ai-crawlers-2026-checklist\/\">Our own AI-crawler checklist<\/a> covers the permission side, and our <a href=\"\/tools\/robots-txt-generator\/#ai-crawlers\">robots.txt generator<\/a> writes the file; this post covers the curation side. Getting one right without the other is only half the job.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Four lines that help answer engines, and four that only look useful \u2014 a plain read of a spec that&#8217;s still being figured out in public.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[],"class_list":["post-1003","post","type-post","status-publish","format-standard","hentry","category-playbook"],"hitmind_lang":"en","hitmind_translations":{"en":1003,"zh":1033},"_links":{"self":[{"href":"https:\/\/hitmind.ai\/blog\/wp-json\/wp\/v2\/posts\/1003","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/hitmind.ai\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/hitmind.ai\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/hitmind.ai\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/hitmind.ai\/blog\/wp-json\/wp\/v2\/comments?post=1003"}],"version-history":[{"count":2,"href":"https:\/\/hitmind.ai\/blog\/wp-json\/wp\/v2\/posts\/1003\/revisions"}],"predecessor-version":[{"id":1397,"href":"https:\/\/hitmind.ai\/blog\/wp-json\/wp\/v2\/posts\/1003\/revisions\/1397"}],"wp:attachment":[{"href":"https:\/\/hitmind.ai\/blog\/wp-json\/wp\/v2\/media?parent=1003"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/hitmind.ai\/blog\/wp-json\/wp\/v2\/categories?post=1003"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/hitmind.ai\/blog\/wp-json\/wp\/v2\/tags?post=1003"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}