Most robots.txt files on the internet were last touched before any of the AI crawlers existed. That means the default answer to “does GPTBot get in?” on the average site isn’t a decision — it’s silence, and silence gets read as “allowed” by some crawlers and “blocked” by others, depending on how they interpret a missing rule.
Our October 2026 study of the top 10,000 domains puts a number on that silence: 71.6% of the robots.txt files we could read don’t name a single AI crawler.
The three lanes
Every AI crawler falls into one of three intents, and the right robots.txt answer is different for each:
- Answer engines that cite you — ChatGPT’s search crawler and browsing agent, Perplexity’s crawler, Claude’s search crawler. Blocking these trades away exactly the citations that are becoming a real referral channel. (Google’s AI Overviews are built from the regular Googlebot crawl, so there’s no separate Google crawler to allow or block here.)
- Training crawlers — bots that scrape to build or fine-tune a model, with no promise of ever sending you a click or a citation. This is where most sites actually want to make a choice, not default into one.
- Everything unlabeled — new agents show up monthly, and a robots.txt written once in 2023 has no opinion on any of them.
What we actually run on hitmind.ai
Our own robots.txt allows everything except /app/, the signed-in product (User-agent: * / Allow: / / Disallow: /app/), then spells out GPTBot, ChatGPT-User, PerplexityBot, Perplexity-User, ClaudeBot, Claude-User, Google-Extended, and Applebot-Extended as explicitly allowed — not because the wildcard needed the help, but because “allowed, written down” beats “allowed by omission” for a product that argues other sites should write theirs down too. We haven’t blocked an AI crawler on hitmind.ai yet; if we ever do, it’ll be a deliberate line in this same file, not a silent gap.
Our free robots.txt tool does exactly this: paste your current file and it evaluates it the way a crawler does, reporting each of twenty AI crawlers as allowed, blocked, or never named. (The five-dimension site audit checks that a robots.txt exists and is readable; the per-crawler verdicts live in the tool.)
The part templates get wrong
The generic “AI-safe robots.txt” templates circulating online tend to block everything with “bot,” “AI,” or “GPT” in the name, on the theory that blocking is the safe default. It isn’t — it’s just a different kind of exposure, trading citation opportunity for a vague sense of control. The right default is a decision, made per crawler, not a regex.
The second trap has nothing to do with templates. A crawler that finds a group with its own name ignores User-agent: * entirely, so a friendly User-agent: GPTBot / Allow: / quietly unblocks every path you blocked for everyone else. In our study of the top 10,000 domains, 29.8% of the robots.txt files that block paths for everyone and name AI crawlers had at least one gap like this, most often on site search, login and API paths. That’s why each named group in our own file repeats Disallow: /app/. If you write named groups, repeat your Disallow lines in each one, or leave the crawler out and let it fall back to *.
HitMind’s free audit checks that your robots.txt exists and is readable, as part of its five-dimension check — the per-crawler verdicts come from the free robots.txt generator and checker, which is live now, alongside the rest of the free tools.
