By hitmind.ai ｜ Published 2026-11-09 ｜ Updated 2026-11-09
On October 12 we fetched /robots.txt from the top 10,000 domains on the Tranco list. The headline result: of the 1,105 files that block paths for every crawler and also name an AI crawler, 329 (29.8%) let that crawler into the blocked paths.
The cause is one rule in RFC 9309. A crawler looks for the group that names it and falls back to User-agent: * only if there is none. So a file that adds a GPTBot group, even one that only says Allow: /, has told GPTBot to ignore every rule under *.
That number held up, within its limits: about a fifth of the domains refused our crawler, mostly bot protection, and aren’t counted. Getting there took five bugs, two wrong denominators and one public correction. This post is that list, because the mistakes are more reusable than the result.
Some context: HitMind is an SEO tool for WordPress sites.
1. Gaps that weren’t there
A trial run on the top 1,000 found 69 files with the gap. We read eight by hand; six were false. The checker took each Disallow pattern under *, such as */search/* or /n$, and asked the named group about it as if it were a URL. A "URL" that doesn’t start with / matches no rule, so even a named group with Disallow: / seemed to let it through. Now each pattern is first turned into a concrete path it must match, and we confirm that * really blocks that path.
Next came named groups that open a specific path on purpose with an Allow rule of their own. That’s a decision, not an oversight, so a gap now counts only when the named group has no rule for the path, or only a blanket Allow: /. The top-1,000 count went from 69 to 36 to 33. Our public checker had the same bug.
2. Eighty-eight asterisks
On the full 10,000 the scan froze three times. The first time, it sat for 40 minutes. We added a 90-second watchdog per site. It froze again, and the watchdog never fired.
Our first theory blamed Node’s fetch behind a proxy. Its evidence, a process at 0% CPU with no connections, turned out to be the parent shell.
The real cause: one file had an Allow rule built from 88 asterisks and a short path, and our matcher compiled each * into .* in a regular expression. Against a path that doesn’t match, that is catastrophic backtracking, and because the regex runs synchronously, the event loop never got a turn to run the watchdog’s timer. We replaced it with a linear wildcard matcher (worst case: path length times pattern length) and replayed 706,000 verdicts from 1,697 stored files through both versions. Zero differences. Three pathological files couldn’t be compared: the old version never finished them.
3. 12,308 Disallow lines
The third freeze was a 494 KB file with 12,308 Disallow lines in one * group. To confirm that * blocks each path, the gap check re-ran the whole * group for every line: about 150 million comparisons. Now it returns at once when no crawler is named, checks only Allow rules at least as long as the Disallow, and stops at a work budget, marking the result truncated. On 2,257 stored files, results were identical before and after.
The structural fix mattered more. The scanner now only fetches and stores raw files; a separate step recomputes every verdict from them. A slow file can’t stall the crawl, and fixing a bug means re-running the analysis, not re-crawling 10,000 domains.
4. A timing test that passed on a laptop
A guard test required an oversized file, 20 named groups of 12,000 rules each, to finish in under five seconds. It took 1.9 seconds on an M3 laptop and failed the production build on a machine at least 2.7 times slower. The real problem wasn’t the test: the checker runs synchronously in the visitor’s browser, and ordinary devices are slower still. We cut the work budget from 20 million comparisons to 5 million. The heaviest real file needs about 2.14 million, so none is truncated, and recomputing the study reproduced the published data byte for byte. A truncated check now says how much it covered instead of reporting "No inheritance gaps".
5. Two denominators
Before publishing, a re-check of the definitions moved the headline. The first draft divided by 1,219 files that "have a * group and contain a Disallow line". That included 114 files that can’t have the gap: their Disallow lines sit only under named groups, or their * group has only an empty Disallow: or a whole-site Disallow: /. The right denominator is files whose * group blocks some path other than /: 1,105. Same 329 files, so 27.0% became 29.8%. The analysis now exits if a file outside the denominator has a gap.
The WordPress comparison was circular: the scanner took "robots.txt has WordPress’s default lines" as evidence of WordPress, and then we measured how many WordPress sites keep those lines. Now only the homepage HTML decides; the 22 sites identified by robots.txt alone count on neither side.
6. The bug we published
Nine days after launch, re-running numbers for a video, we found that "28.4% of files name at least one AI crawler" was wrong. The analysis recorded how each crawler matched by the first letter of named, wildcard or none. "None" starts with N too, so a file with no group for a crawler and no * group counted as naming it. The corrected share is 23.2%: 1,283 of 5,539 files, not 1,574. Twenty-three fields changed, all "named" counts. No blocking figure moved, and the headline didn’t either: its denominator requires a * group, so "none" can’t occur. The page carries a dated correction, and the script now refuses to start unless that case is coded explicitly.
What we’d keep
- Store raw responses; compute everything from them in a separate step.
- Write the denominator down first, and fail when a case lands outside it.
- Don’t classify with the variable you’re about to measure.
- Spell categories out instead of deriving them from a first letter.
- Diff old and new code on real files before shipping a fix.
- Put corrections on the page, with a date.
If you maintain a robots.txt
If an AI crawler’s group exists only to welcome it, delete the group; the crawler falls back to *. If it needs rules of its own, repeat every Disallow line from * inside its group. Our free checker lists every path a named crawler can reach that * blocks. It runs in your browser and uploads nothing: https://hitmind.ai/tools/robots-txt-generator/#check
The study, method and CSV: https://hitmind.ai/research/ai-crawler-access-2026/
Disclosure: we make HitMind. The checker is free.