Crawlsonar

Agent Readiness Checker

The web is getting a new kind of visitor: AI agentsthat don’t just read a page, they discover endpoints, call APIs and act. This checks whether your site exposes the agent-web discovery signals those agents look for.

Probes the origin-level discovery surface AI agents use: llms.txt, MCP, OpenAPI/API-catalog, OAuth discovery, Markdown negotiation, Link headers, security.txt and AI-bot rules.

Page-reading vs agent-ready

Our AI Readiness Checker looks at how well an LLM can read a page. This tool looks one level up: can an autonomous agent discover and use your site as a service? These are emerging standards — most sites score low today, which is exactly the early-mover opportunity.

What we probe

  • llms.txt — a curated map of your key pages for AI tools, at the site root.
  • AI-crawler rules — an explicit allow/block policy for GPTBot, ClaudeBot, Google-Extended… in robots.txt.
  • Content Signals — a Content-Signal: directive in robots.txt declaring how your content may be used (search / ai-input / ai-train).
  • Markdown content negotiation — does the site serve clean text/markdown when an agent sends Accept: text/markdown?
  • DNS-based agent discovery (DNS-AID) — ServiceMode SVCB/HTTPS records at _index._agents / _a2a._agents so an agent can find your endpoints from DNS alone (ideally DNSSEC-signed).
  • MCP endpoint — a Model Context Protocol manifest (e.g. /.well-known/mcp.json) so agents can call your tools directly.
  • OpenAPI / API catalog — a machine-readable API description (/openapi.json, /.well-known/api-catalog).
  • OAuth discovery — /.well-known/oauth-authorization-server / oauth-protected-resource metadata for authenticated agent access.
  • Discovery Link headers — Link: headers pointing agents at your llms.txt, API description or service docs.
  • security.txt — the standard machine-readable contact/trust file at /.well-known/security.txt.

We only report signals we can actually observe over HTTP/DNS — no guesses. Newer, still-settling proposals (agentic-commerce and bot-auth schemes) aren’t scored until they’re observable in the wild.

DNS for AI Discovery (DNS-AID)

Most discovery signals live over HTTP at a /.well-known/ path — but an agent that only knows your domain can find your service endpoints straight from DNS. DNS-AID publishes well-known entrypoint records under _agents.<domain> — an _index._agentsdiscovery index and per-protocol endpoints like _a2a._agents (Agent-to-Agent) — as ServiceMode SVCB/HTTPS records (RFC 9460) carrying alpn and endpointparameters. Sign the discovery zone with DNSSEC so validating resolvers return authenticated data an agent can trust. Example:

_index._agents.example.com. 3600 IN HTTPS 1 . alpn="h2" endpoint="agents.example.com"

We query these records over DNS-over-HTTPS and report whether they resolve and whether the answer is DNSSEC-authenticated. It’s an emerging standard — see the IETF DNS-AID draft and RFC 9460 (SVCB/HTTPS).

Where to start

The quickest wins are llms.txt and an explicit AI-crawler policy — both are one file each and both feed how AI answer engines treat you. Build them with our llms.txt Generator and AI robots.txt Generator.

Related tools

Frequently asked questions

▸ How is this different from the AI Readiness Checker?

AI Readiness scores how well an LLM can read a single page's content. Agent Readiness scores the origin-level discovery surface autonomous agents use — llms.txt, MCP, OpenAPI, OAuth discovery, Markdown negotiation, Link headers and security.txt.

▸ My score is low — is that bad?

Not necessarily. These are emerging agent-web standards, so most sites score low today. It's an early-mover opportunity: llms.txt and an explicit AI-crawler policy are quick wins that also help AI answer engines.

▸ Do you test agentic-commerce or bot-auth standards?

Only signals we can actually observe over HTTP/DNS are scored. Very new proposals aren't included until they're reliably detectable, so the score stays honest.