How to control AI crawlers (GPTBot, ClaudeBot and more)
Allow or block AI crawlers deliberately. Here's what each bot does, how to set an explicit robots.txt policy, and how to verify it's actually working.
AI companies crawl the web for two reasons: to train models and to answer live questions (AI search). You can allow or block each, per bot, in your robots.txt — but only if you do it on purpose. Many sites accidentally block bots they'd want, or allow ones they'd rather not.
Who's who#
| User-agent | Operator | Purpose |
|---|---|---|
| GPTBot | OpenAI | Model training |
| OAI-SearchBot | OpenAI | ChatGPT search index |
| ChatGPT-User | OpenAI | Fetches a page a user asked about |
| ClaudeBot | Anthropic | Model training |
| Claude-User | Anthropic | Fetches a page a user asked about |
| PerplexityBot | Perplexity | Answer-engine index |
| CCBot | Common Crawl | Open dataset used for training |
| Google-Extended | Gemini training opt-out token (not a live crawler) |
Google-Extended and Applebot-Extended aren't crawlers — they're opt-out tokens. A Disallow for them opts you out of that AI training without affecting search indexing.Set a policy in robots.txt#
A common, balanced policy: block training crawlers, allow AI search so you stay citable in answers.
# Block model-training crawlers
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
# Allow AI search / answer engines
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /Generate a full, commented policy for every known AI crawler with the AI robots.txt Generator.
Verify it's working#
Policy on paper isn't proof. Check your server access log with the AI Crawler Log Analyzer to see which AI bots actually visited and whether they got a 200 (allowed) or 403 (blocked). A bot you meant to allow getting 403s means a robots or WAF rule is quietly making you invisible.
Frequently asked questions
▸ ▾ Does blocking GPTBot remove my content from ChatGPT?
It stops future training crawls, but content already in the training data remains, and it doesn't affect live retrieval by OAI-SearchBot or ChatGPT-User unless you block those too. Blocking is forward-looking and partial.
▸ ▾ Will blocking AI crawlers hurt my SEO?
No — GPTBot, ClaudeBot, CCBot and Google-Extended are separate from the search crawlers (Googlebot, Bingbot). Blocking the AI ones doesn't affect classic search ranking. Just don't accidentally block the search bots.
▸ ▾ Is robots.txt enough to block a crawler?
Only for well-behaved bots that choose to honour it. For hard enforcement, block by user-agent or verified IP at your server, CDN or WAF.
Do it with a free tool
Related guides
Published 2026-07-09 · Updated 2026-07-09 · By Crawlsonar.