AI crawlers in robots.txt: a study of the top 10,000 sites (2026)

Name: AI crawler access in the top 10,000 domains (October 2026)
Creator: HitMind

Key findings

29.8% of the robots.txt files that block paths for everyone and name an AI crawler let that crawler past at least one of those blocks (329 of 1,105). In 182 of them, the crawler's group only allows everything: a blanket Allow: / or an empty Disallow:.
The paths most often let back in: site search (155 files), login and account pages (136), APIs (92), admin areas (79) and cart or checkout pages (70).
The crawlers let through most often: GPTBot (187), OAI-SearchBot (179), ChatGPT-User (178), PerplexityBot (150), ClaudeBot (139).
28.4% of robots.txt files name at least one AI crawler. GPTBot is blocked from the whole site by 14.8% (21.6% among the top 1,000 domains); CCBot (15.3%) and Bytespider (15.1%) are blocked most often.
20.0% block at least one AI training crawler, but only 7.1% do so while leaving all 9 AI answer engines open.
14.4% of the sites that answered serve a valid llms.txt; 2.7% of robots.txt files carry Cloudflare's Content-Signal line.
WordPress sites name AI crawlers less often than other sites (18.0% against 28.2%).

The rule behind it

A robots.txt file is a list of groups. Each group starts with one or more User-agent lines and lists the rules for those crawlers. The standard, RFC 9309, has a crawler look for the group that names it. Only if there is none does it fall back:

“If no matching group exists, crawlers MUST obey the group with a user-agent line with the "*" value, if present.”
RFC 9309, section 2.2.1

Google documents the same thing for its own crawlers: user agent specific groups and the global * group are not combined. So the moment a file adds a group for GPTBot — even one that only says Allow: / to welcome it — GPTBot stops reading every rule written under User-agent: *.

FIGURE 1 One file, two readings

robots.txt, as written

User-agent: *
Disallow: /search
Disallow: /account/
Disallow: /api/

User-agent: GPTBot
Allow: /

What each crawler may fetch

Path	Every other crawlere.g. Googlebot	GPTBothas its own group
`/search?q=…`	Blocked	Allowed
`/account/settings`	Blocked	Allowed
`/api/v2/orders`	Blocked	Allowed
`/blog/any-post`	Allowed	Allowed

Highlighted rows: paths the file blocks for everyone except the crawler it names.

Two ways to fix it

If the group only exists to welcome the crawler (it says nothing but Allow: /), delete it. The crawler falls back to User-agent: * and follows the same rules as everyone else.
If the crawler needs rules of its own, repeat every Disallow line from User-agent: * inside its group:

User-agent: GPTBot
Disallow: /search
Disallow: /account/
Disallow: /api/

Our free robots.txt checker finds this for you: paste a file and it lists every path a named crawler can reach that User-agent: * blocks.

How we count it: a file has a gap when a path is blocked under User-agent: * and a named AI crawler's own group has no rule for that path at all, or only a blanket Allow: /. If the group allows a specific path on purpose (for example Allow: /api/public/), we don't count it. Blocking the whole site under * while allowing named crawlers is a common deliberate pattern, so Disallow: / on its own doesn't count either.

What gets let back in

None of these paths is secret — robots.txt is public, and it isn't access control. But they are the paths site owners block on purpose: search results that spawn endless near-duplicate URLs, account pages, and endpoints that cost money to serve.

FIGURE 2 Types of path let back in Share of the 329 files with a gap. A file can count in several types.

Path type	Share of files	Files
Site search results	47.1%	155
Login, account and sign-up pages	41.3%	136
API endpoints	28.0%	92
Admin areas	24.0%	79
Cart, checkout and orders	21.3%	70
Feeds and XML files	13.4%	44

Grouped by patterns in the blocked path (search, ?s=, login, account, /api/, admin, cart, checkout, feed and similar), so the types are approximate.

The most common shape is a group that only allows everything — a blanket Allow: / or an empty Disallow:: 182 of the 329 files (55.3%). The other 147 give the crawler rules of its own but leave out some of the paths blocked for everyone else.

FIGURE 3 Crawlers let past the blocks Number of the 329 files with a gap in which each crawler is let through. Top 10 of 20.

Crawler	Files	Operator
`GPTBot`	187	OpenAI
`OAI-SearchBot`	179	OpenAI
`ChatGPT-User`	178	OpenAI
`PerplexityBot`	150	Perplexity
`ClaudeBot`	139	Anthropic
`Google-Extended`	137	Google
`CCBot`	84	Common Crawl
`Amazonbot`	65	Amazon
`Claude-SearchBot`	64	Anthropic
`Claude-User`	63	Anthropic

Google-Extended is a permission token, not a crawler: Google fetches with Googlebot, which still follows its own group. For it, a gap is a gap in the rules, not necessarily in what gets fetched.

Who blocks which AI crawler

Share of the 5,539 robots.txt files we could parse whose rules block each crawler from the whole site — the verdict for /. That includes files that block every crawler under User-agent: *: 238 files (4.3%) do. The top 1,000 domains block more than the rest: 21.6% block GPTBot, against 14.1% for ranks 1,001–10,000.

FIGURE 4 Blocked from the whole site, by crawler Share of parseable robots.txt files: all 5,539, the top 1,000 (495 files) and ranks 1,001–10,000 (5,044).

Crawler	Used for	Blocked · all	Top 1,000	1,001–10,000
`CCBot`Common Crawl	Training	15.3%	23.8%	14.5%
`Bytespider`ByteDance	Training	15.1%	24.8%	14.2%
`GPTBot`OpenAI	Training	14.8%	21.6%	14.1%
`ClaudeBot`Anthropic	Training	13.8%	21.4%	13.0%
`Google-Extended`Google	Training	12.1%	19.0%	11.4%
`meta-externalagent`Meta	Training	12.1%	20.0%	11.3%
`Applebot-Extended`Apple	Training	11.6%	18.2%	10.9%
`Amazonbot`Amazon	Training	11.4%	16.8%	10.9%
`PerplexityBot`Perplexity	Answer engine	11.0%	18.4%	10.3%
`ChatGPT-User`OpenAI	Answer engine	9.5%	14.9%	9.0%
`DuckAssistBot`DuckDuckGo	Answer engine	8.3%	14.5%	7.7%
`meta-externalfetcher`Meta	Other	8.2%	13.9%	7.7%
`Perplexity-User`Perplexity	Answer engine	8.1%	15.8%	7.3%
`OAI-SearchBot`OpenAI	Answer engine	8.0%	12.9%	7.5%
`Claude-User`Anthropic	Answer engine	7.8%	15.2%	7.0%
`Claude-SearchBot`Anthropic	Answer engine	7.6%	14.3%	6.9%
`Google-CloudVertexBot`Google	Other	7.5%	15.2%	6.7%
`Amzn-SearchBot`Amazon	Answer engine	5.7%	10.5%	5.2%
`Amzn-User`Amazon	Answer engine	5.6%	10.1%	5.2%
`OAI-AdsBot`OpenAI	Other	4.4%	9.1%	4.0%

Used for, from each vendor's documentation: training crawlers collect pages to build models; answer engines fetch pages to answer a question and can cite them; other covers ad checks and similar fetchers. Perplexity says Perplexity-User generally ignores robots.txt because a user asked for the page, so blocking it states a preference it won't enforce.

Block training, keep answers: 7.1%

A common recommendation is to split the decision: refuse training crawlers such as GPTBot and CCBot, which give nothing back, and allow answer engines such as OAI-SearchBot and PerplexityBot, which can cite and link to you. Few sites do it. Of the 1,106 files that block at least one training crawler, 714 (64.6%) also block at least one answer engine. Only 392 (7.1% of all files) block training while leaving all 9 answer engines open.

FIGURE 5 Training crawlers vs. answer engines Share of the 5,539 parseable robots.txt files. Blocked means blocked from the whole site.

Pattern	Share of files	Files
Block at least one training crawler	20.0%	1,106
Block at least one answer engine	13.1%	727
Block training, leave every answer engine open	7.1%	392
Block all 20	3.5%	194

8 training crawlers, 9 answer engines and 3 others. Blocking all 20 is almost never done by name: 193 of the 194 files that block all 20 do it by blocking every crawler.

llms.txt and Content Signals

llms.txt, proposed in September 2024, is a Markdown file at the root of a site that gives language models a summary and a list of key pages. Its one required element is a # title line. 878 of the 6,082 sites that answered our request (14.4%) serve a valid one: status 200, plain text rather than a web page, starting with a title line. The top 1,000 domains are a little ahead: 18.5%, against 14.0% for ranks 1,001–10,000.

FIGURE 6 What /llms.txt returned Share of the 6,082 sites that answered. Excludes 403, 429, 5xx and network errors.

Response	Share of sites	Sites
No file (404 or 410)	70.6%	4,296
Valid llms.txt	14.4%	878
An HTML page instead of a file	6.7%	409
Redirected to a different page	4.3%	264
Plain text without a # title line	2.3%	142
Other responses	1.5%	93

Content Signals, published by Cloudflare in September 2025, add a Content-Signal line to robots.txt with three yes-or-no preferences: search, ai-input and ai-train. 150 of the 5,539 files (2.7%) carry one, and 90 of those say ai-train=no. The line states a preference; it is not part of RFC 9309, and crawlers are not required to read it.

WordPress sites

We identified 396 WordPress sites from their homepage HTML. 41.3% of their robots.txt files still contain WordPress's default rules (Disallow: /wp-admin/ and Allow: /wp-admin/admin-ajax.php). They name AI crawlers less often than other sites and block GPTBot about as often.

FIGURE 7 WordPress and other sites Sites whose homepage we could read (not a bot challenge). Robots.txt rows use files we could parse.

	WordPress	Other sites
Sites	396	4,894
Parseable robots.txt	361	3,959
Name at least one AI crawler	18.0%	28.2%
Block GPTBot from the whole site	10.0%	10.5%
Block at least one training crawler	12.5%	15.1%
Serve a valid llms.txt	16.6%	14.7%

WordPress: a generator tag, an api.w.org link, or both wp-content and wp-includes paths in the homepage HTML. The 22 sites with WordPress rules in robots.txt but no trace in the homepage are left out of both columns. WordPress is uncommon this high up the list (7.5% of readable homepages), so this is a small sample.

Check your own file

Paste your robots.txt into our free checker. It shows what Googlebot, Bingbot and 20 AI crawlers may fetch, and lists every path a named crawler can reach that User-agent: * blocks. It runs in your browser; nothing is uploaded.

Open the robots.txt checker

Writing a new file? The generator on the same page copies your rules into every crawler group it adds, so none of them has a gap.

FAQ

Does User-agent: * apply to GPTBot?

Only when no group in the file names GPTBot. As soon as a group starts with User-agent: GPTBot, GPTBot follows that group alone and ignores every rule under User-agent: *. Among the top 10,000 domains, 29.8% of the robots.txt files that block paths for everyone and name AI crawlers let a named crawler past at least one of those blocks.

How many websites block GPTBot?

14.8% of the 5,539 robots.txt files we could read among the top 10,000 domains block GPTBot from the whole site: 11.4% by naming it, the rest by blocking every crawler. Among the top 1,000 domains the share is 21.6%.

How do I block AI training but allow AI search?

Give each training crawler you want to refuse (for example GPTBot, ClaudeBot, CCBot and Google-Extended) its own group with Disallow: /. Leave answer engines such as OAI-SearchBot, Claude-SearchBot and PerplexityBot out of the file so they follow User-agent: *, or give each one a group that repeats every rule from User-agent: *. Only 7.1% of the sites in this study block training while leaving every answer engine open.

How many sites have an llms.txt file?

14.4% of the sites that answered our request serve a valid llms.txt (878 of 6,082). Among the top 1,000 domains it is 18.5%.

Is robots.txt enough to keep AI crawlers out?

No. robots.txt is a request. Well-behaved crawlers follow it; others don't, and some fetchers that act for a user say they don't apply it — Perplexity says so for Perplexity-User. Anything that has to stay private needs a login or a server-side rule.

Method

Domains: the top 10,000 entries of the Tranco list, a ranking of popular domains built for research (list 4672X, generated 2026-10-11).
When: one pass on 2026-10-12, 07:35–09:34 UTC.
Crawler: HitMindBot, user agent Mozilla/5.0 (compatible; HitMindBot/1.0; +https://hitmind.ai/bot). At most three requests per domain, one after another: /robots.txt, then /llms.txt (first 64 KB), then the homepage (first 256 KB, used only to detect WordPress). We followed each site's rules for HitMindBot, or User-agent: * when it doesn't name us, and skipped what they disallow (270 sites for llms.txt). When robots.txt returned a server error or didn't answer, we treated the site as fully disallowed and fetched nothing else, as RFC 9309 says to.
File size: robots.txt was read up to 500 KiB, the limit Google applies. 10 files larger than 100 KB were analyzed on their first 100 KB.
Rules: RFC 9309 group selection and longest-match precedence (Allow wins a tie), with * and $ wildcards — the same code as our public robots.txt checker. Blocked means the rules disallow / for that crawler.
AI crawlers: the 20 user agents in our robots.txt generator, each with its vendor's documentation, grouped by what the content is used for (8 training, 9 answer engines, 3 other).
llms.txt counted as valid when it returned 200, was not an HTML page, was not redirected to a different path, and its first non-empty line was a # title.

What the 10,000 domains returned for /robots.txt
Served a robots.txt we could parse	5,539	55.4%
403 Forbidden	1,208	12.1%
503 Service Unavailable	920	9.2%
Other 5xx or 429	70	0.7%
No file (404 or 410)	667	6.7%
Other 4xx	61	0.6%
An HTML page instead of a robots.txt	480	4.8%
No answer (DNS, TLS, timeout, refused)	1,054	10.5%
Other	1	0.0%

Limitations

About a fifth of the domains refused us. 1,208 answered 403 and 920 answered 503 for robots.txt, most of them bot protection. They are left out of every share above. Sites that turn away unknown bots may turn away AI crawlers more often too, so the blocking shares are more likely too low than too high.
Many entries are not websites. Popular-domain lists include API, CDN, ad and analytics hostnames that serve no pages, which accounts for many of the 1,054 domains that didn't answer at all.
One snapshot from one location. A few sites serve different files by region or by user agent.
Rules, not behavior. A gap shows what the rules permit, not what any crawler fetched, and some gaps may be deliberate. We apply RFC 9309; we can't see how each vendor's parser behaves.
Permission tokens. Google-Extended and Applebot-Extended are not crawlers. Google and Apple fetch with Googlebot and Applebot, so a gap for either token may change nothing in practice.
Approximations. Path types are pattern matches on the path. WordPress detection depends on the homepage, and 1,573 homepages were bot-challenge pages.

Data and citation

Download the aggregate figures (CSV). Every number on this page comes from that file, with its denominator. We don't publish per-site results, as our crawler page promises.

While running the numbers we found and fixed four problems in our own robots.txt checker — two that reported gaps that weren't there, and two that made it hang on unusual files (one with 88 wildcards in a rule, one with 12,000 Disallow lines). Every figure here was recomputed from the stored files with the fixed code.

Cite as

HitMind, “AI crawler access in the top 10,000 sites,” October 2026. https://hitmind.ai/research/ai-crawler-access-2026/

Questions about the data, or a breakdown we didn't publish: support@hitmind.ai.