How to recognise it
Requests from HitMindBot carry this user agent:
Mozilla/5.0 (compatible; HitMindBot/1.0; +https://hitmind.ai/bot)
The performance part of the free audit works differently: your homepage is loaded once in a headless Chrome browser running Lighthouse. That request identifies as Chrome-Lighthouse, not HitMindBot.
We don't publish a fixed list of IP addresses. Identify us by the user agent.
When it visits
- Someone runs the free audit on your site. One pass over a handful of files, described below.
- Someone runs the orphan page checker on your site. A crawl that follows links from your homepage, at most once every 24 hours per site.
- A site owner connects their own site to a HitMind account. We then crawl that site on their behalf.
- Our research scan, at most once a month. For a report on how sites treat AI crawlers, it reads three public files on sites near the top of a published ranking of popular domains. Details below.
Apart from the research scan, HitMindBot doesn't roam the web: it doesn't follow links to other sites, and it doesn't build its own list of sites to crawl.
The free SEO audit
Each run fetches your homepage, robots.txt, your sitemap, llms.txt and ai.txt, and checks up to 10 links found on your homepage to see whether they are broken. It runs once, when someone asks for it.
It reads your robots.txt in order to report on it. Because it only fetches your homepage and these standard files, a Disallow rule doesn't make it skip them.
The orphan page checker
- Reads
robots.txtand your sitemaps first (at most 30 sitemap files). - Follows links from your homepage to at most 500 pages, then opens up to 200 sitemap pages that nothing linked to, to check that they load.
- Fetches at most 4 pages at a time, and no more than 240 requests a minute to one site.
- Crawls each site at most once every 24 hours. Anyone who runs the check again within that window gets the saved result; nothing new is fetched.
- Only follows ordinary links (
<a href>) on your domain and its subdomains. It doesn't submit forms, log in or run JavaScript.
It obeys robots.txt: any path disallowed for HitMindBot — or, if there is no HitMindBot group, for all crawlers (*) — is not fetched.
The research scan
- Which sites: domains near the top of the Tranco list, a public ranking of popular domains built for research. We don't pick sites by hand.
- What it fetches:
robots.txtfirst, thenllms.txtand the first 256 KB of your homepage, which we use only to tell whether the site runs WordPress. At most three requests per site, one at a time. - robots.txt: always read first, because that is how it learns your rules.
llms.txtand the homepage are skipped if robots.txt disallows them for HitMindBot — or, if there is no HitMindBot group, for all crawlers (*). - How often: at most once a month.
- What we publish: aggregate numbers only, such as the share of sites that block a given AI crawler. We don't publish your file or a list of sites.
Results of the October 2026 scan: AI crawler access in the top 10,000 sites.
How to limit or block it
To keep the orphan page checker out of part of your site, add a group to robots.txt:
User-agent: HitMindBot
Disallow: /private/
Use Disallow: / to keep it out of the whole site. The research scan follows the same group: it will still read robots.txt itself, which is how it learns your choice, and then stop.
Because the free audit only fetches your homepage and a few standard files, robots.txt doesn't stop it. To block HitMindBot completely, block its user agent at your server, firewall or CDN — it identifies itself on every request.
Contact
If HitMindBot is causing you problems, email support@hitmind.ai with your domain and what you saw in your logs.