Is your site blocking ChatGPT?
One line in robots.txt can keep your pages out of ChatGPT, Claude, Perplexity and AI Overviews. Enter a domain for a red/green matrix of all 13 AI crawlers. No sign-up.
How sites block AI crawlers by accident
Almost nobody sets out to block ChatGPT. It happens when a security plugin ships an AI-blocking preset, when a staging Disallow: / survives a launch, or when a CDN adds bot rules that a marketing team never sees. The file below allows everything except one crawler — and most teams would never notice which one.
User-agent: * Allow: / User-agent: GPTBot Disallow: / Sitemap: https://acme.com/sitemap.xml
A named block always wins over User-agent: *, so this site is open to every crawler on the internet except OpenAI’s. The matrix above makes that visible in one screen.
What each crawler is for
Blocking is a legitimate choice — but it should be a decision, not an accident. Here is what each one actually costs you.
GPTBotOpenAIBlocking removes your pages from the corpus OpenAI trains and grounds on.
OAI-SearchBotOpenAIPowers the ChatGPT search index. Blocking it removes you from ChatGPT search results.
ChatGPT-UserOpenAIFetches a page when a ChatGPT user follows a link. Blocking it breaks live citations.
ClaudeBotAnthropicAnthropic’s crawler. Blocking it keeps your content out of Claude’s knowledge.
Claude-WebAnthropicLegacy Anthropic web fetcher. Still honoured by some deployments.
anthropic-aiAnthropicOlder Anthropic token. Sites that block only this one often still allow ClaudeBot.
PerplexityBotPerplexityPerplexity cites sources inline. Blocking it forfeits a high-intent citation surface.
Google-ExtendedGoogleControls Gemini / AI Overviews grounding only. It does NOT affect normal Google Search ranking.
Applebot-ExtendedAppleApple Intelligence training opt-out. Does not affect Siri/Spotlight search indexing.
CCBotCommon CrawlCommon Crawl feeds many open models and datasets. A block propagates widely.
cohere-aiCohereCohere’s crawler for retrieval and model grounding.
meta-externalagentMetaMeta AI / Llama crawler. Blocking it removes you from Meta AI answers.
BytespiderByteDanceByteDance/Doubao crawler. Widely blocked for aggressiveness — a deliberate block is defensible.
Crawlers can reach you. Can they find the right pages?
An open robots.txt gets an agent through the door. An llms.txt tells it which pages are worth reading. Check yours next — also free, also no sign-up.
Validate your llms.txtAI crawler questions
Which crawlers does this check?
All 13 AI user agents Growzo tracks: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-Web, anthropic-ai, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, cohere-ai, meta-externalagent, Bytespider. These are the exact tokens the crawlers send, so the check reflects what your robots.txt actually says about them.
How does it decide allowed vs blocked?
It parses your robots.txt with the same engine our site audit uses, applying Google’s documented longest-match semantics: the most specific matching rule wins, and an Allow beats a Disallow of equal length. A block naming a crawler explicitly overrides the site-wide User-agent: * rule.
I have no robots.txt. Is that bad?
For AI access, no. The robots exclusion standard treats a missing file as no restrictions, so every crawler is allowed — the tool reports that as the result rather than as an error. You may still want one for other reasons, such as declaring your sitemap.
Does blocking Google-Extended hurt my Google rankings?
No. Google-Extended controls only whether your content can be used for Gemini and AI Overviews grounding. It does not affect Googlebot or classic Search ranking. Applebot-Extended works the same way for Apple Intelligence.
Does allowing these crawlers guarantee I get cited?
No, and be suspicious of anyone who says otherwise. Allowing them is necessary, not sufficient — it removes a hard blocker. Whether a model then cites you depends on your content and how often it is referenced elsewhere. This tool only tells you whether the door is open.
Do crawlers actually obey robots.txt?
Compliance is voluntary. Major operators state they honour it, but this tool reports what your file instructs, not what any crawler did. If you need enforcement rather than a request, block at the CDN or WAF by user agent and IP.
Unblocking them is the floor, not the goal.
Growzo watches your crawler access, publishes the content models want to cite, and tracks whether ChatGPT, Claude and Perplexity actually name you.
Open-source · Self-host · BYOK · ₹10k/mo hard cap