Crawlsonar

Agent Readiness Checker

The web is getting a new kind of visitor: AI agentsthat don’t just read a page, they discover endpoints, call APIs and act. This checks whether your site exposes the agent-web discovery signals those agents look for.

Probes the origin-level discovery surface AI agents use: llms.txt, MCP, OpenAPI/API-catalog, OAuth discovery, Markdown negotiation, Link headers, security.txt and AI-bot rules.

Page-reading vs agent-ready

Our AI Readiness Checker looks at how well an LLM can read a page. This tool looks one level up: can an autonomous agent discover and use your site as a service? These are emerging standards — most sites score low today, which is exactly the early-mover opportunity.

What we probe

  • llms.txt — a curated map of your key pages for AI tools, at the site root.
  • AI-crawler rules — an explicit allow/block policy for GPTBot, ClaudeBot, Google-Extended… in robots.txt.
  • Markdown content negotiation — does the site serve clean text/markdown when an agent sends Accept: text/markdown?
  • MCP endpoint — a Model Context Protocol manifest (e.g. /.well-known/mcp.json) so agents can call your tools directly.
  • OpenAPI / API catalog — a machine-readable API description (/openapi.json, /.well-known/api-catalog).
  • OAuth discovery — /.well-known/oauth-authorization-server / oauth-protected-resource metadata for authenticated agent access.
  • Discovery Link headers — Link: headers pointing agents at your llms.txt, API description or service docs.
  • security.txt — the standard machine-readable contact/trust file at /.well-known/security.txt.

We only report signals we can actually observe over HTTP/DNS — no guesses. Newer, still-settling proposals (agentic-commerce and bot-auth schemes) aren’t scored until they’re observable in the wild.

Where to start

The quickest wins are llms.txt and an explicit AI-crawler policy — both are one file each and both feed how AI answer engines treat you. Build them with our llms.txt Generator and AI robots.txt Generator.

Related tools

Frequently asked questions

▸ How is this different from the AI Readiness Checker?

AI Readiness scores how well an LLM can read a single page's content. Agent Readiness scores the origin-level discovery surface autonomous agents use — llms.txt, MCP, OpenAPI, OAuth discovery, Markdown negotiation, Link headers and security.txt.

▸ My score is low — is that bad?

Not necessarily. These are emerging agent-web standards, so most sites score low today. It's an early-mover opportunity: llms.txt and an explicit AI-crawler policy are quick wins that also help AI answer engines.

▸ Do you test agentic-commerce or bot-auth standards?

Only signals we can actually observe over HTTP/DNS are scored. Very new proposals aren't included until they're reliably detectable, so the score stays honest.