Blog · Sep 19, 2026
2% of websites block AI crawlers in robots.txt
About 1 in 47 small business sites blocks at least one AI crawler in robots.txt. Here is the breakdown.
Key takeaways
- 219 of 10,369 scored sites (2.1%) block at least one AI crawler in robots.txt.
- 152 sites (1.5%) block three or more, and 67 (0.6%) block one or two.
- Blocking is a legitimate choice for some sites, but a leftover or copied rule can make the choice for you.
The numbers
Our scanner checks each site's robots.txt against ten AI crawlers and scores crawler access in three steps.
| Crawler access score | What it means | Sites | Share of 10,369 |
|---|---|---|---|
| 100 | None of the ten blocked | 10,150 | 97.9% |
| 40 | One or two blocked | 67 | 0.6% |
| 0 | Three or more blocked | 152 | 1.5% |
So 219 sites (2.1%) block at least one. These are blocks in robots.txt, so they are separate from the 752 sites whose servers refused our scanner outright. See that post.
What a blocking rule looks like
A rule in robots.txt names a crawler and tells it to stay out:
`` User-agent: GPTBot Disallow: / ``
That asks OpenAI's training crawler to stay away from the whole site. Whether that is the right choice depends on which crawler it is: training crawlers, search crawlers and user-initiated fetchers do different jobs, and blocking one doesn't block the others. See robots.txt and AI crawlers for the differences.
How blocks happen, and how to check
Most of these rules arrive by accident: a WordPress setting left on, a security plugin default, a copied template or a "temporary" block that stayed. Our post on blocking AI crawlers by accident covers the causes and gives a 60-second check.
Should you block AI crawlers?
There are good reasons to: a publisher protecting paywalled content, a business with licensing limits on its text, or an owner who doesn't want content used to train models. If you opt out, do it deliberately: name the specific crawler, limit the rule to the paths that need protection, and write down why. For most local businesses, being readable by the crawlers behind AI search is what keeps you eligible to be cited, so a blanket block is rarely the right call.
What to do
- Open
yoursite.com/robots.txtand look for AI crawler names next toDisallow, and forUser-agent: *followed byDisallow: /. - Remove rules you didn't choose, and keep the ones you did.
- Re-scan to confirm your crawler access score.
Send to your web person: > Please review our robots.txt for rules blocking AI crawlers and confirm which ones are intentional. Remove any blanket Disallow that we didn't choose.
About this data: 10,369 small business sites scored in October 2026. Crawler access is scored against ten AI crawlers: 100 if none are blocked in robots.txt, 40 if one or two, and 0 if three or more.