The AI crawler blocking index
How many sites tell GPTBot, ClaudeBot, PerplexityBot and Google-Extended to stay out? We read a fixed list of public robots.txt files on a schedule and publish what they say — with the sample size, the method, and the raw rows attached.
No crawl has run yet.
This index publishes measurements, so with zero runs on record there is nothing to show — and we would rather show you an empty page than a placeholder chart. The first run will appear here with its full sample size and methodology attached.
In the meantime you can run the same check against your own domain, for free and without signing up.
Check your robots.txtMethodology
Read this before quoting any number above. The sample is small and fixed, which makes it reproducible but not representative.
What the sample is
A committed seed list of well-known public sites across news, retail, developer, reference and SaaS categories. It is a fixed sample, not a ranking, not traffic-weighted, and not "the web". Its purpose is like-for-like comparison across runs.
What we fetch
One request to /robots.txt per domain, serially, with a pause between domains. Redirects are followed but re-validated at every hop, non-public addresses are refused, and the response is size- and time-capped.
How a verdict is decided
The same parser our site audit uses, applying Google’s documented longest-match semantics: the most specific matching rule wins, an Allow beats a Disallow of equal length, and a group naming a crawler explicitly overrides User-agent: *. Verdicts are evaluated for the site root.
Missing versus unreachable
A 404 or 410 means there is no robots.txt, and the standard treats that as no restrictions — recorded as allowed, because that is what crawlers will do. A timeout, DNS failure, 403 or 5xx means we learned nothing, and is recorded as unknown.
How percentages are computed
Per crawler: blocked domains divided by domains with a readable robots.txt for that crawler. Unknown rows are never in the numerator or the denominator. A denominator of zero is reported as a dash, never as 0%.
What this does not measure
Whether any crawler obeys the instruction. robots.txt compliance is voluntary and cannot be verified from outside. This index measures stated policy, and CDN- or WAF-level blocking is invisible to it.
Where does your own site sit?
This page measures other people’s robots.txt. Run the identical check against yours — same parser, same 13 crawlers, free and no sign-up.
Check your robots.txtAbout this index
How big is the sample?
Every number on this page states its own denominator, because the sample is small and fixed rather than large and representative. It is a committed seed list of well-known public sites across news, retail, developer, reference and SaaS categories. It is not "the top N sites on the web", it is not traffic-weighted, and it should never be read as a measurement of the web as a whole.
What happens when a robots.txt cannot be read?
It is recorded as unknown and excluded from every percentage, then reported separately as an unreachable domain. A timeout, a 403, a 5xx or a DNS failure tells us nothing about a site’s policy, so we refuse to guess. The one case we do treat as a real verdict is a 404: the robots exclusion standard says a missing file means no restrictions, and that is what crawlers act on.
Why is a single run not shown as a trend?
Because one observation has no direction. Change figures appear only once a bot has at least two measured runs, and each series contains only points we actually observed — nothing is interpolated, smoothed or carried forward to fill a gap.
Does this measure whether crawlers obey the rules?
No. robots.txt compliance is voluntary and unobservable from the outside. This index measures what site owners instruct, which is the only thing anyone can verify from public data. If you need enforcement rather than a request, block at the CDN or WAF by user agent and IP.
Can I use these numbers?
Yes — that is why the raw CSV is published alongside the chart. Cite the run date and the sample size, both of which are on this page and in the file. If you recompute the percentages and get a different answer, we would rather hear about it than not.
Does blocking Google-Extended hurt Google rankings?
No. Google-Extended governs only Gemini and AI Overviews grounding; it does not affect Googlebot or classic Search ranking. Applebot-Extended works the same way for Apple Intelligence. That distinction matters when reading this index, because those two bots are often blocked deliberately.
Being crawlable is the floor, not the goal.
Growzo watches your crawler access, publishes the content models want to cite, and tracks whether ChatGPT, Claude and Perplexity actually name you.
Open-source · Self-host · BYOK · ₹10k/mo hard cap