Most robots.txt files on the internet were last touched before any of the AI crawlers existed. That means the default answer to “does GPTBot get in?” on the average site isn’t a decision \u2014 it’s silence, and silence gets read as “allowed” by some crawlers and “blocked” by others, depending on how they interpret a missing rule.<\/p>\n
Our October 2026 study of the top 10,000 domains<\/a> puts a number on that silence: 71.6% of the robots.txt files we could read don’t name a single AI crawler.<\/p>\n Every AI crawler falls into one of three intents, and the right robots.txt answer is different for each:<\/p>\n Our own Our free robots.txt tool<\/a> does exactly this: paste your current file and it evaluates it the way a crawler does, reporting each of twenty AI crawlers as allowed, blocked, or never named. (The five-dimension site audit checks that a The generic “AI-safe robots.txt” templates circulating online tend to block everything with “bot,” “AI,” or “GPT” in the name, on the theory that blocking is the safe default. It isn’t \u2014 it’s just a different kind of exposure, trading citation opportunity for a vague sense of control. The right default is a decision, made per crawler, not a regex.<\/p>\n The second trap has nothing to do with templates. A crawler that finds a group with its own name ignores HitMind’s free audit checks that your GPTBot, ClaudeBot, PerplexityBot and nine more \u2014 who to allow, who to block, and why the default template gets it wrong.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[],"class_list":["post-1004","post","type-post","status-publish","format-standard","hentry","category-playbook"],"hitmind_lang":"en","hitmind_translations":{"en":1004,"zh":1024},"_links":{"self":[{"href":"https:\/\/hitmind.ai\/blog\/wp-json\/wp\/v2\/posts\/1004","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/hitmind.ai\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/hitmind.ai\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/hitmind.ai\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/hitmind.ai\/blog\/wp-json\/wp\/v2\/comments?post=1004"}],"version-history":[{"count":3,"href":"https:\/\/hitmind.ai\/blog\/wp-json\/wp\/v2\/posts\/1004\/revisions"}],"predecessor-version":[{"id":1396,"href":"https:\/\/hitmind.ai\/blog\/wp-json\/wp\/v2\/posts\/1004\/revisions\/1396"}],"wp:attachment":[{"href":"https:\/\/hitmind.ai\/blog\/wp-json\/wp\/v2\/media?parent=1004"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/hitmind.ai\/blog\/wp-json\/wp\/v2\/categories?post=1004"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/hitmind.ai\/blog\/wp-json\/wp\/v2\/tags?post=1004"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}The three lanes<\/h2>\n
\n
What we actually run on hitmind.ai<\/h2>\n
robots.txt<\/code> allows everything except \/app\/<\/code>, the signed-in product (User-agent: *<\/code> \/ Allow: \/<\/code> \/ Disallow: \/app\/<\/code>), then spells out GPTBot, ChatGPT-User, PerplexityBot, Perplexity-User, ClaudeBot, Claude-User, Google-Extended, and Applebot-Extended as explicitly allowed \u2014 not because the wildcard needed the help, but because “allowed, written down” beats “allowed by omission” for a product that argues other sites should write theirs down too. We haven’t blocked an AI crawler on hitmind.ai yet; if we ever do, it’ll be a deliberate line in this same file, not a silent gap.<\/p>\nrobots.txt<\/code> exists and is readable; the per-crawler verdicts live in the tool.)<\/p>\nThe part templates get wrong<\/h2>\n
User-agent: *<\/code> entirely, so a friendly User-agent: GPTBot<\/code> \/ Allow: \/<\/code> quietly unblocks every path you blocked for everyone else. In our study of the top 10,000 domains<\/a>, 29.8% of the robots.txt files that block paths for everyone and name AI crawlers had at least one gap like this, most often on site search, login and API paths. That’s why each named group in our own file repeats Disallow: \/app\/<\/code>. If you write named groups, repeat your Disallow<\/code> lines in each one, or leave the crawler out and let it fall back to *<\/code>.<\/p>\nrobots.txt<\/code> exists and is readable, as part of its five-dimension check \u2014 the per-crawler verdicts come from the free robots.txt generator and checker<\/a>, which is live now, alongside the rest of the free tools<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"