AI robots.txt Generator
Decide which AI crawlers may read your site and get an explicit robots.txt policy. Start from a preset, fine-tune per bot, copy and publish.
Keep your content out of model-training datasets while staying citable in AI search and answer engines. A common balance. 4 allowed · 6 blocked
# AI crawler policy — generated by Crawlsonar # Review, then merge into the robots.txt at your site root: https://example.com/robots.txt # Note: robots.txt is advisory — only well-behaved crawlers honour it. # Google-Extended and Applebot-Extended are opt-out tokens (not live crawlers); a Disallow opts you out of that AI training. # OpenAI — model training User-agent: GPTBot Disallow: / # Anthropic — model training User-agent: ClaudeBot Disallow: / # Common Crawl — open dataset used for training User-agent: CCBot Disallow: / # ByteDance — model training User-agent: Bytespider Disallow: / # Google — Gemini training opt-out token User-agent: Google-Extended Disallow: / # Apple — AI training opt-out token User-agent: Applebot-Extended Disallow: / # OpenAI — ChatGPT search index User-agent: OAI-SearchBot Allow: / # Perplexity — answer engine index User-agent: PerplexityBot Allow: / # OpenAI — fetches a page a user asked about User-agent: ChatGPT-User Allow: / # Anthropic — fetches a page a user asked about User-agent: Claude-User Allow: /
Merge this into the robots.txt at your site root, then confirm it with the AI Readiness Checker.
Why set an explicit AI policy?
AI crawlers fall into three groups with different trade-offs. An explicit policy is a deliberate governance signal — leaving it to the default looks unconsidered either way.
- Training / dataset crawlers (GPTBot, ClaudeBot, CCBot, Bytespider) collect content to train models. Blocking them keeps your content out of training sets; it does not affect whether you rank in Google.
- AI search & answer engines (OAI-SearchBot, PerplexityBot) index pages to cite them in answers. Allowing them can win you referral traffic and citations.
- User-triggered fetchers (ChatGPT-User, Claude-User) fetch a specific page because a user asked about it. Blocking these can frustrate your own visitors.
Two honest caveats: robots.txt is advisory — only well-behaved crawlers obey it; it is not access control. And Google-Extended / Applebot-Extended are opt-out tokens, not real crawlers: a Disallow opts you out of that AI training without affecting normal search crawling.
FAQ
Does blocking GPTBot hurt my Google ranking?
No. GPTBot is OpenAI’s crawler; it has nothing to do with Googlebot or your Google rankings. To control Google’s Gemini training specifically, use the Google-Extended token.
Will this stop AI from using my content entirely?
No tool can. robots.txt is honoured by compliant crawlers only. It is the standard, expected signal — but not an enforcement mechanism.
Where do I put the output?
Merge these User-agent groups into your existing robots.txt at https://example.com/robots.txt — do not create a second file.
Related tools
- AI Readiness Checker — see your current AI crawler policy and confirm the new one after publishing.
- llms.txt Generator — the complementary "here’s my best content" file.
Last reviewed: 2026-07-09. Crawler list and token behaviour current at that date; AI crawlers change often — re-check periodically.
Frequently asked questions
▸ ▾ Does robots.txt actually stop AI crawlers?
Only well-behaved crawlers honour robots.txt. For hard enforcement, block by user-agent or verified IP at your server, CDN or WAF.
▸ ▾ What is Google-Extended?
It's an opt-out token, not a live crawler. Disallowing it opts you out of Gemini training without affecting Google Search indexing.