Have an account?Sign in
The file most sites need: one rule for every crawler, the folders to keep out, and where your sitemap is. Then, if you want, decide on twenty AI crawlers one at a time — and check what your current file actually permits.
Runs entirely in your browser. Nothing is uploaded, nothing is stored.
For most sites this is the whole file: a default, the paths to keep out, and your sitemap. All of it goes into one User-agent: * group that every crawler reads.
Crawlers may read every page except the paths you list below.
One per line, starting with /. A trailing slash covers the whole folder; * matches anything and $ marks the end. A missing slash or a full URL is fixed for you.
Pages inside a blocked path that should stay open. The longer rule wins, so Allow: /wp-admin/admin-ajax.php beats Disallow: /wp-admin/.
Seconds between requests. Google ignores this line — its documentation lists crawl-delay as unsupported — while Bing reads it. Leave it off unless a crawler is straining your server.
One per line, full URL with https://. WordPress's built-in sitemap is /wp-sitemap.xml; Yoast and Rank Math use /sitemap_index.xml. A blog on a separate system needs its sitemap listed too.
Without a group of their own, AI crawlers follow the rules above like every other crawler. Name them here only to treat some differently. There is no answer that fits every site — only what each crawler does, and whether you want that.
Off — AI crawlers follow the rules above. Pick a preset or change any row to turn it on.
Blocks the training crawlers, keeps the ones that can cite you. · Also removes you from AI answers that would have linked back.
Everything starts allowed because that is what hitmind.ai's own robots.txt says — our choice, not a recommendation for your site. Change any row.
These read a page to answer a question and link back to it. Blocking one trades away the citation, which is the part that sends people to your site.
These collect pages to build or fine-tune a model. No promise of a click or a citation in return. This is where most sites actually want to make a choice rather than default into one.
Ad safety checks, social card previews, R&D fetches. They neither train on your pages nor cite them, so blocking one costs little and gains little.
Replace the whole of robots.txt with this, or merge the groups into what you already have.
# robots.txt — generated by https://hitmind.ai/tools/robots-txt-generator
# All crawlers
User-agent: *
Allow: /
Where it goes — The root of the domain, always: https://example.com/robots.txt. Not a subfolder, not a subdomain you also own — each hostname reads its own file.
Paste your current robots.txt. Every rule is applied the way a crawler applies it — Google and Bing first, then each AI crawler.
Across the top 10,000 domains, 29.8% of the files that block paths for everyone and name AI crawlers have at least one gap of this kind. Read the study →
The homepage by default. Try a folder you meant to block, such as /wp-admin/.
This reads rules, not enforcement. robots.txt is a request; a crawler that ignores it is stopped at the server or CDN, not here.
Paste a file above to see what it permits.
robots.txt looks like a list read top to bottom. It is not.
It finds the group matching its own name and uses that. If there isn't one, it uses User-agent: *. It never combines the two — so a path blocked only under * is open to every crawler you named elsewhere.
Disallow: /a/ plus Allow: /a/b/ means /a/b/c is allowed: the longer rule wins regardless of order. When two rules are the same length, Allow wins.
Disallow: with nothing after it is the opposite of Disallow: / — it permits everything. Both appear in real files, usually by accident.
Well-behaved crawlers obey it. The rest read it and continue. Anything you genuinely need kept out belongs behind auth or a server rule, not in this file.
A disallowed URL can still show up in Google: its documentation says Google 'may still index the URL and show it in search results without a snippet.' To keep a page out of search, let it be crawled and mark it noindex.
Sitemap: applies to every crawler wherever it sits in the file, and must be a full URL, protocol and host included. It is the one line not scoped to a User-agent.
The free audit checks robots.txt, your sitemap, llms.txt and the homepage, scores five dimensions, and ranks what to fix first. No account, no card.
5 dimensions · no signup · ~60 seconds