The Crawlsonar bot

Crawlsonar fetches public web pages to run diagnostics — AI-readiness, security headers, SEO and DNS/email checks — either when someone asks us to check a site, or for our public readiness leaderboard. We aim to be a good citizen: we identify ourselves, respect robots.txt, and cryptographically sign our requests.

User-Agent

Crawlsonar/1.0 (+https://crawlsonar.com/bot; diagnostic scan)

Purpose

On-demand and benchmark diagnostics of public pages. We read server-rendered HTML only — no JavaScript execution, no logins, no form submissions.

Respects robots.txt

Yes. Our automated scanner honours Disallow rules for the Crawlsonar token and the * group, and won’t crawl a site that disallows us.

Signed requests

Every request is signed with Web Bot Auth (HTTP Message Signatures, RFC 9421). Verify against our public key directory: /.well-known/http-message-signatures-directory.

Rate

Light and bounded — a small number of requests per host, with timeouts. We do not hammer origins.

JavaScript / cookies

None. We don't run JS, store cookies, or attempt to bypass anti-bot challenges — if a site blocks us, we back off.

Allow or block Crawlsonar

To block us, add this to your robots.txt— we’ll comply:

User-agent: Crawlsonar Disallow: /

To allow us through a WAF or bot-management rule, match our User-Agent token Crawlsonar, or verify our Web Bot Auth signature. On Cloudflare, a WAF rule that skips bot protection when http.user_agent contains “Crawlsonar”works; the cryptographic signature above is the tamper-proof way to confirm it’s really us.

Contact

Questions, or want us to stop crawling your site? Email [email protected] and we’ll act promptly. Crawlsonar is operated by Northstar Infinity Works Ltd.