robots.txt Checker
Validate a site’s robots.txt: catch accidental full-site blocks, missing sitemap directives and your AI crawler policy — with issues and fixes.
What this tool checks
- Full-site block — an accidental
Disallow: /that hides the whole site. - Sitemap directive — whether robots.txt points crawlers at your sitemap.
- AI crawler policy — an explicit stance on GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot and more.
Important: robots.txt controls crawling, not indexing. It does not remove already-indexed URLs from search — use a noindex meta tag or header for that. And it is advisory: only well-behaved crawlers obey it.
Related tools
- AI robots.txt Generator — build an explicit AI crawler policy.
- Sitemap Checker — validate the sitemap this file points to.
Last reviewed: 2026-07-09.
Frequently asked questions
▸ ▾ What does this checker flag?
Full-site blocks, a missing sitemap directive and your AI-crawler policy (GPTBot, ClaudeBot and others), with fixes.
▸ ▾ Is robots.txt enough to block a crawler?
Only for well-behaved bots that choose to honour it. For hard enforcement, block by user-agent or verified IP at your server, CDN or WAF.