Crawlsonar

Changelog

134 releases since 9 July 2026, newest first. Each entry says what changed and why — and what was verified, and what wasn’t. Running version: v1.19.1.

v1.19.1

Added

  • OPERATIONS.md (repo root — /docs/ is deliberately uncommitted in this repo) — deploy procedure, the production env-var table with what each gates, the after-every-deploy checklist (health version, the mysentry "Send test heartbeat" click that registers all three cron monitors, the 401-not-404 cron spot-check, deployment cleanup) and the list of owner tasks code cannot do (legal read of /terms, real testimonials, competitor re-verification, Stripe promo code, Supabase migrations/passkeys). Written so the go-live steps live in the repo instead of a chat transcript.

Verification

  • Smoke-tested the production build (Next 16.3.3) locally: /, /pricing, /alternatives, /terms, /tools, /leaderboard, /api/health, /llms.txt all 200; MCP tools/list 200; llms.txt serves the new Product section.

v1.19.0

Changed — zero-debt pass: dependencies, lint, types, tracing, discovery

  • Dependencies patched to npm audit 0 vulnerabilities (was 8: 2 moderate, 6 high): Next.js 16.2.10 → 16.3.3 (brings sharp 0.35.4, closing the libvips CVEs bundled with Next's image optimizer) and the postcss override 8.5.16 → 8.5.26 (the 8.5.16 pin was itself a security fix in 0.93.0; its version has since been flagged). Full test suite and production build pass on the new versions.
  • ESLint and tsc are now completely clean (before: 18 errors + 9 warnings, 2 test-file type errors). Fixed across ~20 files: unescaped quotes/apostrophes in JSX; an <a href="/leaderboard"> swapped for <Link>; seven components deferring their initial state-setting effect a tick (the set-state-in-effect rule); five tool clients whose useCallback read the input state instead of its own argument (fixing the exhaustive-deps warnings without behaviour change — the argument is what actually ran); four unused imports/vars; the two long-standing test-file type errors (fixture cast, union narrowing).
  • Output tracing fully scoped again under Next 16.3.3: the new tracer pulls the whole project into any route whose store calls fs.readdir on an env-overridable path — without the build warning 16.2.10 printed. Fixed in the monitor store, the reminders store and the admin dashboard's report counter (literal .data/... paths; the reminders test now stubs process.cwd() like the testimonials test). Verified: 0 routes above 100 traced src files, down from 9 × 500.
  • llms.txt gains a Product section (pricing/tracking, agencies, alternatives, leaderboard, learn, API & MCP, terms) so AI readers see the product, not only the free tools.
  • .env.example documents the last undocumented knobs (NEXT_PUBLIC_GA_ID, INBOX_INGEST_SECRET, SCAN_LOG_SALT, PDF_FONTS_DISABLED, the test-only store-dir overrides) and drops the now-dead MONITOR_STORE_DIR/REMINDER_STORE_DIR entries.

Verification

  • Measured: npm audit 0 vulnerabilities; npx eslint src 0 problems; npx tsc --noEmit 0 errors; 593 tests pass; production build clean; nft-trace scan shows 0 routes >100 src files.

v1.18.0

Changed — the price anchor is now a verified fact, terms are accepted at sign-in, crons report as themselves

  • Competitor price anchor verified at the source (fetched from otterly.ai/pricing on 23 August 2026): Otterly Standard is $189/month ($160/month billed annually) for 100 tracked prompts on four engines, with more engines as add-ons. The pricing page and /alternatives now quote that figure with the check date instead of the earlier unsourced "$189–199 for 100 prompts" range. Peec's page publishes no figures in plain text; /alternatives reports what third-party reviews say (Starter ≈ $95/50 prompts, Pro ≈ $245/150 prompts, three engines included) and labels it as such. Re-check both before any future copy change — prices move.
  • Terms acceptance where it matters: the sign-in form states that continuing means agreeing to the Terms of Service and Privacy Policy (links), and the WebAPI JSON-LD termsOfService now points at /terms rather than /privacy.
  • Every revenue-relevant cron is its own mysentry monitor: the daily tracking cron (<base>-tracking) and the daily lifecycle cron (<base>-lifecycle) now send heartbeats with their error state and processed counts, alongside the hourly monitor cron. The admin "Send test heartbeat" button pings all three, which registers them on mysentry from one click — closing the gap where a heartbeat-registered monitor cannot notice a cron that never started. The admin card says to press it after any deploy that adds a cron.
  • Testimonial supply: the weekly digest for paid accounts ends with a one-line request for a sentence we may quote (name and company) — the honest way to fill the testimonials module added in 1.17.0. Free-tier digests keep the upgrade line instead.

Verified

  • tsc clean apart from the two pre-existing test-file errors, eslint clean on every touched file, full test run green, production build clean with no whole-project tracing.
  • Production observed from outside on 23 August 2026: /api/health reports version 1.6.0 — every release from 1.7.0 to this one is unshipped; /api/v1/cron/lifecycle returns 404 there (not deployed), the pricing page shows plans (billing enabled). The Vercel CLI is not installed on this machine, so production environment variable names could not be listed.

Still outside code

  • Legal review of /terms; real customer quotes (the digest now asks); production env values (Resend, Supabase public vars, engine keys) — verify in the Vercel dashboard; press "Send test heartbeat" in Admin → Dashboard after the deploy; a Stripe promotion code for WINBACK_PROMO_CODE if win-back should carry a discount. Deployment only on explicit request.

v1.17.0

Added — terms, a buyer's checklist, social proof, cohorts, MCP on a key

  • Terms of Service at /terms (footer link, sitemap, WebPage + breadcrumb JSON-LD): operator facts as on /privacy, what the free tools and paid plans are, acceptable use, trials and automatic renewal, the 14-day refund the pricing FAQ already promises, content ownership, a plain statement that AI-visibility numbers are sampled observations and not guarantees, liability caps, termination, England & Wales law. Written to match what the product does today; needs a legal read before it is relied on — the checkout and sign-in flows do not yet reference it.
  • /alternatives — the checklist for choosing an AI visibility tracker and where Crawlsonar stands. Eight questions to ask any vendor, a fact table read from the same plan definitions the app enforces (engines, samples per prompt, metrics, competitors, alerts, white-label, price per tracked prompt, day-one experience, free plan), the existing price anchor ("Otterly and Peec list roughly $189–199/month for 100 prompts, at the time of writing"), and an honest "where we are not the right fit" box (no rank tracking, no SSO/SAML, no social listening, no guarantee of being recommended). No competitor feature claims beyond the price anchor — those would need verification. Linked from the pricing page and the homepage pricing section; FAQ JSON-LD; ISR hourly via the cached catalogue.
  • Testimonials module — src/lib/testimonials.ts (KV document testimonials:v1, file locally; cached public read with tag testimonials), admin API /api/v1/admin/testimonials (GET/POST/PATCH/DELETE, admin only, validated), an "Social proof" panel under Admin → Users to add, publish/hide and remove quotes, and a <Testimonials /> section on the homepage and pricing page that renders nothing until the first quote is published. No placeholder quotes anywhere: the module is the form for the real ones.
  • Cohorts, churn and LTV under Admin → Revenue: subscriptions grouped by the month they started, how many are still live, retention %, live MRR per cohort; 30-day cancellation rate over (live + churned); LTV as ARPU ÷ monthly churn, labelled as a rule-of-thumb until there are months of data. Pure computeCohorts with tests.
  • MCP accepts the customer API key — /api/mcp with Authorization: Bearer <key> is metered on the key's daily quota (JSON-RPC -32001 with HTTP 401/429 when invalid or exhausted) instead of the anonymous 30-messages-per-minute-per-IP limit; authenticated traffic is attributed to the key in Admin → API & MCP and on Account → API. Documented on /api-docs, the MCP discovery GET and the admin limits card. Anonymous access is unchanged.
  • Annual saving spelled out on the pricing cards: "billed $X/yr — 2 months free (save $Y)".

Verification

  • Measured on the dev server: /terms, /alternatives, /pricing, /about render 200 with the expected titles; the pricing page contains no testimonial section while the list is empty; annual sub-copy shows "save $68" (Pro) and "save $238" (Growth); GET /api/mcp advertises optional auth; anonymous tools/list returns 16 tools; a bogus bearer key returns the JSON-RPC error with HTTP 401; the admin testimonials API returns 401 unauthenticated.
  • tsc clean (two pre-existing test-file errors unchanged), eslint clean on all touched files, 593 tests pass, production build below.

Not done in code (needs the owner)

  • Legal review of /terms; real customer quotes; verification of any competitor claim before adding one; production env checks (Resend, billing flag, Supabase public vars, engine keys); MySentry registration of the lifecycle cron; Stripe promotion code for WINBACK_PROMO_CODE. Deployment is not performed without an explicit request.

v1.16.0

Added — win-back, GA4 mirroring, a README that says what this is

  • Win-back email about a month after a cancellation (30–44-day window, once per subscription, honours unsubscribes): the projects and history are still there on the free plan, here's the door — with WINBACK_PROMO_CODE quoted when set (create the code in Stripe; checkout accepts promotion codes). Runs from the daily lifecycle cron alongside trial reminders and lead nurture.
  • Browser events are mirrored to GA4 (gtag('event', …) when the tag is loaded after consent), so the GA4 property sees the same scan / report / bridge / capture steps as the first-party store. Server-recorded conversions (signup, trial_started, upgrade, cancel) stay server-side by design; the GA4 admin tab now says exactly that instead of claiming events it never received.
  • README replaces the create-next-app boilerplate: what the product is and for whom, the map of the codebase, how to run it, the conventions (version = changelog = commit; dormant-until-configured; privacy-minimal; server-recorded conversions) and a five-minute post-deploy check.

Verification

  • 589 tests (+1), tsc and eslint clean. The win-back send itself is not exercised locally (no Resend); the window logic is unit-tested.

v1.15.0

Added — agencies, about, changelog, 970 leaderboard pages, Performance in Learn

  • /agencies — the Agency plan as a landing page: 25 client projects, 400 prompts, all five engines daily, 10 competitors, 10 seats, white-label PDF reports, the price from the live catalogue (revalidated hourly), "what a client sees each month", a FAQ with FAQPage JSON-LD, the trial CTA and a walkthrough e-mail link.
  • /about — what the product does, how it scores, what data it touches, who runs it (company number, registered office) and how to reach a person. AboutPage JSON-LD.
  • /changelog — every release rendered from CHANGELOG.md at build time (a small parser and a safe inline renderer, unit-tested): the public proof of pace. Linked in the footer together with About and For agencies.
  • /leaderboard/{domain} — one static page per site in the snapshot (970): composite, AI-readiness and security scores with a plain-English band, rank overall and within the category, category neighbours as internal links, what the scores mean, "check your own site" and the tracking bridge; blocked sites get the bot-protection explanation instead. ItemPage + breadcrumb JSON-LD, listed in the sitemap (1,063 URLs now); the leaderboard table links to them.
  • Learn — the empty Performance category gets core-web-vitals and caching-and-ttfb (beginner and professional explanations, fix steps, FAQ), with issue-id rules so performance findings link to them.
  • The sitemap gains the two tools it was missing (/tools/blacklist-check, /tools/domain-expiry-reminder) plus the new pages.

Fixed — two things the production build measured

  • The homepage and /agencies rendered dynamically (Cache-Control: private, no-cache) despite revalidate = 3600: the KV client fetches with cache: "no-store", and any page that reads the catalogue during render inherits that. Public pages now read it through unstable_cache (hourly, tag billing-catalog, invalidated with revalidateTag(…, "max") when the admin saves prices). Measured on next start: both now answer x-nextjs-cache: HIT, s-maxage=3600.
  • Five functions shipped the entire project — /r/[token], its OG image, the PDF route, the leads and reports APIs carried ~700 traced files each, ~490 of them project source. Found by diffing the routes' .nft.json; cause: fs.readdir(storeDir()) in the report store's sweep — a readdir on a runtime-computed path makes Turbopack's tracer give up and include everything. Pre-existing since the store was written. Fixed by reading the literal .data/reports directory (and never sweeping under test). The PDF route now traces 193 files, no route exceeds 100 source files, and the build warning "whole project was traced unintentionally" is gone. The PDF fonts moved to new URL(…, import.meta.url) asset references (emitted as server assets and listed in both PDF routes' traces; the outputFileTracingIncludes TTF globs are no longer needed) — verified on the production build: a freshly generated report embeds Geist, not the Helvetica fallback.

Verification

  • 588 tests; tsc clean; production build: 1,141 static pages, zero warnings. On next start: /about, /agencies, /changelog, /leaderboard/github.com (scored), /leaderboard/aa.com (blocked), the two Learn entries and the sitemap render; an unknown domain 404s; a shared report's PDF embeds Geist. Nothing in this release needs a runtime service.

v1.14.0

Added — the customer relationship now has emails, and the lead list is worked

  • Daily lifecycle cron GET /api/v1/cron/lifecycle (vercel.json, 10:00 UTC, CRON_SECRET): (1) trial-ending reminders 3 days and 1 day before the first charge — plan, date, amount and the portal link (src/lib/lifecycle/trials.ts; for a trialing Stripe subscription current_period_end is the trial end); (2) lead nurture — three emails to people who downloaded a PDF report, on day 1 (the tool-specific fix that moves their score most, linking their report), day 3 (who AI recommends instead → free account) and day 7 (track it weekly → plans). One email per address per run, steps in order, never twice (markers), never to leads older than 14 days — so the day this ships does not blast the historical list (src/lib/lifecycle/nurture.ts).
  • Unsubscribe for lead email (/leads/unsubscribe): the link carries the address and an HMAC (LEADS_UNSUB_SECRET, falling back to CRON_SECRET), GET shows a confirmation page, POST implements RFC 8058 one-click; every nurture and waitlist email carries the link and the List-Unsubscribe headers. Opt-outs are markers keyed by a hash of the address.
  • Waitlist announcement from Admin → Users → Waitlist: "{plan} is open", optional early-adopter promo code, once per address, honouring unsubscribes (src/lib/lifecycle/waitlist.ts, POST /api/v1/admin/waitlist). The email the waitlist was promised and never got.
  • Weekly digest for the free tier too: the headline share of voice, the week's delta and cited sources, plus a plain statement of what Pro adds (competitors, three engines, alerts). Paid tiers unchanged.
  • Leads now store the shared-report token so follow-ups can link the report; one-shot markers live in src/lib/lifecycle/markers.ts (KV with a 13-month TTL, files locally).

Verification

  • 583 tests (+7: unsubscribe tokens and opt-out, nurture scheduling, trial timing); tsc and eslint clean. Live on the dev server: /leads/unsubscribe?token=bad → 400 (GET and POST); /api/v1/cron/lifecycle → both jobs report "skipped":"email not configured" — the expected dormant answer without Resend.
  • Not exercised: actual sends (no Resend locally) and the Stripe currentPeriodEnd semantics during trial — taken from Stripe's documented behaviour. After deploy: register the new cron in MySentry (a heartbeat that never fired can't alert) or run it once by hand, and check Admin → Usage for lifecycle_email events.

v1.13.0

Added — every free tool now leads somewhere

  • <NextStepBridge> under the result on all 49 tool pages, the Domain Audit and every shared report (/r/{token}): AI/SEO tools (readiness, llms.txt, robots, sitemap, schema, SEO, link preview, compare, fix pack, performance, generators for AI/SEO, utilities) bridge to AI-visibility tracking — "This is one check, today. Now see whether AI recommends you — every week" → /account?next=/projects, with /pricing as the secondary link; DNS, email, TLS and security tools (25 pages) bridge to monitoring first — "Fixed it? Get told before it breaks again" → /tools/monitoring, with tracking as the secondary link. The monitoring page itself gets none (it is the product).
  • Clicks are tracked as bridge_clicked {tool, kind, target}, so Admin → Usage shows which tools actually send people into the product — the number the revenue plan was missing.
  • Inserted by a codemod (<NextStepBridge tool="…" kind="…" /> immediately before each page's <ToolFaq>), so the placement is uniform and a page without a bridge is a lint-visible exception, not a forgotten one.

Verification

  • tsc clean, 576 tests pass; the tool pages' 11 eslint findings (react/no-unescaped-entities, react-hooks/exhaustive-deps) are pre-existing — identical count with the change stashed. Rendered /tools/dns-lookup locally: bridge present below the content, above the FAQ.

v1.12.0

Added — the funnel is measured, and the first cycle runs at creation

  • Server-recorded conversion events (src/lib/server-events.ts): signup (first sign-in ever, via a once-per-user marker in KV / a local file), project_created, first_cycle_ready, trial_started, upgrade, payment_failed, cancel — the last four derived from Stripe status *transitions* (previous record vs. incoming), so overlapping webhook events count each milestone once. The browser beacon endpoint no longer accepts signup/upgrade (a beacon can't fake a conversion any more); monitor_create_* (fired by the monitoring UI, rejected with 400 until now) and bridge_clicked / cta_clicked are accepted.
  • Admin → Usage → Funnel, last 30 days: Scans → Leads → Accounts → Projects → First data → Trials → Paid with step-to-step conversion, plus payment failures and cancellations beside it. computeFunnel is pure and unit-tested.
  • First cycle in a minute. Creating a project now suggests the first three prompts from the domain (Claude Haiku; skipped without ANTHROPIC_API_KEY) and runs an instant cycle — 3 prompts × 2 samples × the plan's engines, calls in parallel — before redirecting, so the project page opens with share of voice, winning competitors and cited sources instead of "No cycles yet". Projects that have prompts but no data get a Run the first cycle now button. Guards: owner only, 3 instant runs per account per day, the global TRACKING_DAILY_BUDGET, and plain messages while engine keys are dormant. The runner's per-project loop became runProjectCycle (shared by the cron — sequential as before — and the bootstrap — concurrent): src/lib/tracking/runner.ts, bootstrap.ts.
  • Lifecycle emails (src/lib/lifecycle/emails.ts; dormant until Resend is configured, intent logged): welcome on first sign-in (one job: create the first project), "your first cycle is in" with the numbers (and a plan nudge on Free), payment failed (→ billing portal), cancelled (projects stay on Free, door open). Trial-ending reminders need a daily cron and ship in the next release.
  • /projects?welcome=1 onboarding checklist; the create and run buttons narrate the 20–40 s wait (useFormStatus); both project pages declare maxDuration = 60 for the synchronous action.

Changed — the pricing page promises only what the code grants

  • AI Visibility Free: Claude-sampled, 3 prompts weekly, first cycle in a minute (was "ChatGPT · Claude · Perplexity · Google AIO" and a "public leaderboard rank" that never existed). Growth: all five engines, 5 projects, 2 seats, 365-day history (was "API + MCP access" — a separate product — and "weekly digest for your team", which every paid tier gets). Agency now states 25 projects, 10 competitors, 10 seats. Monitoring: hourly checks (was 5-/1-minute), email + webhook alerts (was status page, SMS credits, white-label status page, team seats, priority alerting). API: daily quotas as enforced — 100 / 500 / 4,000 per day, 2 / 5 / 10 keys (was "10,000 / 100,000 per month" and "webhook results"). Tool counts come from the index.

Verification

  • 576 tests pass (+4), tsc and eslint clean; /pricing rendered locally with the new bullets. Not exercised locally (no Supabase, engine or Resend keys in .env.local): the bootstrap run, the welcome / first-cycle emails and the webhook transitions — reasoned from code and written to fail soft (every branch degrades to the scheduled cron). First check after deploy: create a project on a free account → the page should open with numbers, and Admin → Usage → Funnel should show project_created and first_cycle_ready.

v1.11.0

Added — structured data on every public page (SEO / GEO)

  • Shared builders in src/lib/jsonld.ts — breadcrumbJsonLd (always starting at Home), webPageJsonLd (with isPartOf / publisher references) and graph — so every page links into one entity graph anchored on the /#organization and /#website nodes the layout declares. Same ids everywhere is what lets search and answer engines merge the pieces.
  • Organization enriched with legalName, the Companies House identifier, the registered address, support email + contactPoint, foundingDate and knowsAbout; WebSite gains inLanguage and a description.
  • New nodes: /tools — CollectionPage + an ItemList of all 51 tools as free WebApplications; /pricing — one Product per product line (AI Visibility, Monitoring, API) with an AggregateOffer and an Offer per active plan from the live catalogue (monthly and annual UnitPriceSpecification, free tier included); /leaderboard — Dataset (snapshot date, variables measured, technique) plus a CollectionPage; /api-docs — WebAPI (endpoint, documentation, provider, free-tier offer; MCP named in the description); /bot — WebPage about the crawler as a SoftwareApplication; /privacy — WebPage with dateModified; /guides and /learn — breadcrumbs. Tool pages' WebApplication now carries @id, publisher, isPartOf, isAccessibleForFree and a Home-first breadcrumb (one helper, 49 pages).
  • /guides and /learn switched from raw JSON.stringify scripts to the escaping JsonLd component.

Verification

  • Fetched 13 pages from the dev server and parsed every application/ld+json block: 0 invalid; each page carries the expected types (Organization + WebSite everywhere; SoftwareApplication + FAQPage on /; CollectionPage + ItemList on /tools; Product ×3 on /pricing; Dataset on /leaderboard; WebAPI on /api-docs; TechArticle + FAQPage on a guide; DefinedTerm + FAQPage on a Learn entry; WebApplication + FAQPage on a tool). 3 new unit tests (572 total); tsc and eslint clean.
  • Rich-result eligibility (Product, FAQ, Breadcrumb, Dataset) is Google's call — run the Rich Results Test on /pricing, /leaderboard and a tool page after deploy.

v1.10.0

Changed — API keys belong to accounts; API & MCP traffic visible in the admin panel

  • API keys are issued from the customer account only. The public generator on /api-docs is gone; the page now sends developers to Account → API (sign-in or a free account, then a key). POST /api/v1/keys requires a session (401 otherwise) and enforces the plan's key count (free 2, Dev 5, Scale 10); new GET /api/v1/keys lists the caller's keys with today's usage, and DELETE /api/v1/keys?id= revokes one. Every key now has an owner: records carry userId and a per-user index (apikeys:user:{id} in KV, a JSON file locally) makes list/revoke possible without scanning the store. Keys minted anonymously before this release keep working for requests; they simply belong to nobody and can't be listed.
  • Account → API tab: plan, requests/day per key, active keys vs. the plan limit, a create form (label optional; the key is shown once), each key's usage today / total calls / last used, and Revoke. Free accounts see the API-plan upsell.
  • Admin → "API & MCP" tab (/admin?tab=api): 7- and 30-day totals for the public API and the MCP server, rejected requests split by cause (burst rate-limit, daily quota, auth, malformed), a per-day table, and ranked breakdowns — API tools, API keys (by public id, matching Account → API), MCP tools and JSON-RPC methods. The limits in force are stated on the page: MCP 30 messages/min per IP, no key, no daily quota; API 60 requests/min per IP plus the per-key daily quota (free 100, Dev 500, Scale 4,000); the per-minute limiter is per serverless instance, the daily quota in KV is exact.
  • Counters, not an event log: src/lib/api/metrics.ts keeps one hash per surface per day (metrics:api:YYYY-MM-DD → total, outcome, tool, key, method; 100-day TTL in KV, JSON files locally). Routes record through Next's after() so metering runs once the response is sent and can never fail a request. Unit-tested (+5 tests, 569 total).
  • /api-docs "Limits & errors" now states the real quotas per plan and the per-IP burst cap; the 401 message for a missing key points to Account → API.

Verification

  • tsc and eslint clean on every changed file; 569 tests pass (key ownership, revoke, path-traversal guard on user ids, metrics aggregation). Not exercised locally: the KV branch (hashes + user index) — it uses the same kvHincrby/kvRpushJson helpers already in production for the rate limiter and event stream.

v1.9.0

Added — passkeys (Face ID, Touch ID, Windows Hello, security keys)

  • Sign in with a passkey on /account: a one-tap WebAuthn sign-in next to the magic link, via supabase.auth.signInWithPasskey() (auth-js 2.110, behind its auth.experimental.passkey flag). The same next continuation is honoured, so "Start free" → passkey → /projects. The email input advertises autocomplete="email webauthn" for conditional UI.
  • Passkeys card on Account → Account: list (name, added, last used), add — auth.registerPasskey() then a friendly name via auth.passkey.update() (default "Chrome on macOS"-style from the user agent) — and remove. Clear recovery note: the email link always works, so a lost device is never a lock-out.
  • Plain-English errors for every failure mode (NotAllowedError cancelled/timed out, InvalidStateError already registered, unsupported browser, insecure context, server not enabled, no passkey for this device) in src/lib/auth/passkeys.ts, unit-tested.
  • Dormant until configured: NEXT_PUBLIC_PASSKEYS_ENABLED=1 shows the buttons and enables the client flag. Prerequisite on the Supabase project: passkeys enabled in Authentication settings (the /auth/v1/passkeys/* endpoints must respond) with the relying-party ID set to the site host. Without that the buttons would fail at the first click, which is why the flag defaults off. .env.example documents it.
  • Why first-factor rather than MFA-only: auth-js exposes both; a second factor on top of a magic link adds friction without removing the inbox round-trip, while a passkey replaces it — the faster sign-in is the one that sells.

Verification

  • 5 new unit tests (564 total); tsc and eslint clean. Not verifiable locally (no Supabase project with passkeys in .env.local): the live ceremony, which needs HTTPS on the real host. First thing to check after enabling: register a passkey on Account, sign out, sign in with it.

v1.8.0

Changed — the homepage now sells AI-visibility tracking; the free tools move below the fold

  • Homepage rebuilt around the paid product. Hero: "Is your brand in the answer?" with one primary action (Start tracking free → /account?next=/projects) and a secondary one-shot check. A static product preview (trend chart, share of voice, "who wins the answers", labelled as sample data) replaces the abstract benefit tiles. Then: a proof strip with the real leaderboard numbers (970 sites, median 76/100, 17% blocked), "what you see every week", how it works, the one-shot AI Visibility check (/tools/ai-visibility, now tracked as scan_started {tool:"home-ai-visibility"}), plan cards from the editable billing catalogue (annual per-month price; feature bullets are the entitlements the code actually grants), the "Can't I just ask ChatGPT myself?" objection, and only then the free tools — the search box, six featured tools and "Browse all N tools" — visible to anyone who scrolls, absent from the first screen. FAQ with FAQPage JSON-LD, a closing CTA, and SoftwareApplication JSON-LD carrying Offers from the catalogue. The page is ISR (revalidate = 3600), so an admin price edit reaches the homepage within an hour without a deploy.
  • The header sells an account: "Sign in" + Start free on desktop and mobile. "Domain audit" leaves the header button (it stays on /tools, the homepage and the mobile menu); the IP widget leaves the nav (it stays on /tools/what-is-my-ip).
  • /tools gets the homepage's type-to-find search, and the tool count comes from the index (ALL_TOOLS.length, 51) — one number everywhere instead of "40+ / 50+ / 55".
  • Funnel fixes from the revenue plan: the /pricing Free-plan CTA now creates an account (/account?next=/projects) instead of opening a one-shot tool; /billing/success sends the new customer to Create your first project (/projects?welcome=1) instead of back to the free tools; sign-in carries a same-site next through the magic-link callback, and a signed-in visitor arriving with ?next= is redirected straight there; the email-capture copy no longer promises that monitoring "launches" (it shipped in v0.60).
  • Metadata: homepage title "Crawlsonar — is your brand in the AI answer?" with a tracking-first description; canonical unchanged.

Verification

  • tsc clean for the change (two pre-existing test-file errors untouched), eslint clean on every changed file, 559 tests pass. Dev server: /, /tools and /account?next=/projects render; homepage copy checked section by section; plan prices $28 / $99 / $249 per month (annual) from the seed catalogue.
  • Not verifiable locally: auth and billing are env-gated. With NEXT_PUBLIC_SUPABASE_URL / NEXT_PUBLIC_SUPABASE_ANON_KEY unset the new CTAs land on "Accounts aren't enabled yet" — confirm both are set in production before this goes live, and that NEXT_PUBLIC_BILLING_ENABLED=1 so the plan cards lead to checkout rather than the waitlist modal.

v1.7.0

Changed — PDF reports redesigned from the ground up

  • Lead-magnet PDF (GET /api/v1/reports/{token}/pdf, AI Readiness / llms.txt / Domain Audit) is now a designed document instead of a text dump. Page one: navy cover band with the sonar mark, score ring with grade pill, severity pills, plain-English verdict (with the "capped by a high issue" explanation when it applies), four KPI tiles (issues by severity, checks passed, weakest area or module, AI-crawler rules / scan time / llms.txt status) and the top three priorities. Then: category score bars, issue cards (severity + category/module pill, Impact / Evidence / Fix in a spec-sheet layout, inline code, example code block, clickable "plain-English explainer" link into /learn/{slug}), the AI-crawler policy table with policy pills, "what an AI assistant actually reads" (front-matter and image syntax stripped from the extraction), the full checks list with drawn pass/fail/n-a glyphs and weights, a numbered "what to do next" (re-run link + share link with its expiry) and a closing bridge to AI Visibility monitoring with a UTM-tagged /pricing link. Running header/footer with "Page i of n", PDF outline bookmarks, document metadata.
  • AI-Visibility tracking PDF (GET /api/v1/projects/{id}/report) rebuilt on the same system: headline share-of-voice ring with the change versus the previous cycle, KPI tiles (share of voice, presence, tone, change since the first cycle), a vector trend chart (share of voice + presence), per-engine table for the latest cycle, cycle history with ▲/▼ deltas, the latest cycle's winning competitors and cited sources as ranked bars (the route now passes latestInsights, so the PDF never says less than the project page), tracked prompts, competitor chips, a "how to read this report" note and a dashboard CTA. White-label (Agency) output carries no Crawlsonar name, mark, link or metadata — covered by a test that greps the uncompressed document.
  • Typography: Geist + Geist Mono (SIL OFL 1.1, licence in src/lib/pdf/fonts/) embedded and subsetted per document — a five-page report is ~70–85 KB. If the TTFs cannot be read at runtime the renderer logs {"at":"pdf/fonts",…} and falls back to Helvetica/Courier under the same logical names: a missing asset degrades the look, never the download. next.config.ts force-includes the TTFs (and pdfkit's .afm metrics) into both PDF function bundles.
  • New shared layer src/lib/pdf/{canvas,theme,format,fonts}.ts: measured text, ensure() page-break guards so no card, table, chart or heading straddles a page, tables that redraw their header after a break, ring / bar / line-chart / code-block primitives, pill / KPI / badge composites. Both renderers keep their public signatures (renderReportPdf, renderTrackingPdf), with an optional { compress: false } for tests.
  • Tests: +9 (font embedding and fallback, llms.txt shape, multi-page pagination with embedded Geist and an outline, tracking branded / empty / white-label / dashboard link) — 559 total.

Verification

  • Rendered the nine locally stored sample reports (ukimmigration.law, stripe.com, github.com, faktury.co.uk, example.com across all three report types) plus synthetic tracking data (branded, white-label, empty state) and inspected every page as a PNG. Unit tests pass; tsc is clean for the new files (the two pre-existing tsc errors in core-web-vitals.test.ts and reports/store.test.ts are untouched).
  • Not yet verified on Vercel: that outputFileTracingIncludes ships the TTFs into the function bundles. After the first deploy, download any report and confirm the PDF embeds Geist (font panel, or strings report.pdf | grep Geist). A Helvetica-only PDF means the fallback kicked in and the function logs will show "at":"pdf/fonts".

v1.6.0

Changed — public pricing page now reflects the admin catalogue

  • /pricing reads prices from the editable billing catalogue (KV, seed as default), so changing a plan’s price in the admin Billing tab now updates the public page too — no code edit. Curated feature copy stays in code; only the amount + which paid plans show (deactivated plans are hidden) come from the catalogue. The "Growth is $X/mo" anchor is computed from the live price. Verified: Pro $28 · Growth $99 · Agency $249 (annual/mo) from the seed.

v1.5.0

Added — full system observability in the admin panel

  • Usage tab: unique visitors, scans run, scans/visitor and total events, plus ranked bars for most-used tools, top countries, most-scanned targets and every event type — all from the first-party event stream (no external call).
  • Revenue tab: MRR (annual normalised to monthly), ARR, paying accounts (active/trialing), ARPU, new/churned this month, past-due to recover, and an MRR-by-plan table with share — from the Stripe-synced subscription records.
  • GA4 tab: connection status (Measurement ID), deep links to Realtime / Reports / Search Console, and exactly which events we send — with an offer to wire the GA4 Data API (service account) for embedded live numbers.
  • Admin sidebar reorganised: Dashboard · Usage · Revenue · Users · Billing · GA4 · Errors. New src/lib/admin/insights.ts computes usage + revenue.

v1.4.0

Added — professional customer dashboard

  • Unified customer dashboard with the same left vertical sidebar as the admin panel (Overview / Projects / Subscription / Team / Account), a plan badge + sign-out in the footer, and a responsive top-strip on mobile. /account and /projects now share one shell (CustomerShell).
  • Overview: welcome header + at-a-glance stat cards (projects used/limit, prompts, engines, cadence/history), primary quick actions, a "shared with you" card, and an empty-state nudge to create the first project.
  • Subscription: current plan + what it unlocks, an Upgrade/Change-plan CTA, and a "Manage subscription" button that opens Stripe's secure customer portal for the signed-in user (new authenticated POST /api/v1/billing/portal-account — resolves the Stripe customer from the *verified session* e-mail, so a user can only ever manage their own billing).
  • Team (seats invite/manage) and Account (sign-in details) as dedicated sections. Cognitive-bias-aware copy throughout (anchoring on limits, clear single primary action per view).

v1.3.0

Added — editable product/price catalogue in the admin Billing tab

  • The admin can now edit the plan list and prices directly in the panel (like the polbooks flow): add/remove plans, rename, change monthly price, toggle active — saved to the KV store (billing:catalog), with the code seed as the default until the first save.
  • Price-aware Stripe sync: because Stripe prices are immutable, changing a price now creates a new Price carrying the same lookup_key (transfer_lookup_key) and archives the old one — new checkouts get the new amount, existing subscribers keep theirs (grandfathering). The status table flags each row as Synced / Price changed / Missing / To archive with catalogue-vs-Stripe amounts.
  • Checkout validates the plan against the editable catalogue (not the static list); the stripe:setup CLI and the admin button share one engine (stripe-catalog.ts). Admin API validates every field with zod and never leaks secret values. +2 tests.

v1.2.0

Added — professional admin panel + Stripe product management

  • Admin redesign: the panel now uses a left vertical sidebar (CMS-style) with icons — Dashboard / Errors / Users / Billing — a brand header, identity + sign-out in the footer, and a full-width content area. Replaces the old top tab-strip (admin-tabs.tsx removed). Responsive: the sidebar collapses to a scrollable top bar on mobile.
  • Billing tab (Stripe product management, like the polbooks flow): shows a live readiness banner, a configuration checklist (secret key + test/live mode, webhook secret, pricing-UI flag, catalogue completeness), and a table of every plan × interval with its lookup key, price and Created/Missing status. A "Create products in Stripe" button runs the same idempotent sync as the CLI — right from the admin UI.
  • Shared catalogue engine src/lib/billing/stripe-catalog.ts (getCatalogStatus / syncStripeCatalog) now backs both the admin button and npm run stripe:setup (DRY). New admin API POST/GET /api/v1/admin/billing (behind the admin gate; returns 4xx not 5xx per the CF-swallows-5xx rule; never leaks secret values — booleans only).

v1.1.1

Fixed — free tools showed a misleading "Network error" for slow/blocking targets

  • Root cause: every tool API returned HTTP 5xx (502/504) for *expected, target-caused* failures (the scanned site blocks bots, times out, DNS-fails, or returns 4xx). Cloudflare (in front of crawlsonar.com) replaces any origin 5xx with its own HTML error page, so the browser never received our JSON {error} — res.json() threw and the client fell back to the generic "Network error — please try again."
  • Fix: target/upstream-failure responses now return HTTP 422 (which Cloudflare passes through untouched, like our existing 400/429) across all 25 tool routes + the WHOIS/RDAP, Certificate-Transparency and link-preview libs. The client now sees the real, specific message (e.g. "Could not reach that page — it may block bots or be too slow").
  • Link-preview also: client now parses the response body defensively (a non-JSON edge page can never surface as a bogus "Network error"), the fetch timeout drops 10s → 8s, and it now runs bot-wall detection — a Cloudflare/Akamai/etc. challenge served as HTTP 200 (e.g. "Just a moment…") is reported as "X challenge is blocking our crawler" instead of returning a junk preview scraped from the interstitial.

v1.1.0

Added — Increment J: sentiment (how AI *talks about* you)

  • Each AI answer that mentions the brand now gets a tone score −1..1 from a cheap, deterministic lexicon (positive/negative terms + simple negation flipping) — no per-answer LLM cost, protecting the paid product's margin. analyzeSentiment scores only the sentence(s) naming the brand; analyzeAnswer attaches it; aggregatePrompt exposes avgSentiment; latestInsights averages it per cycle.
  • Migration 0004 adds a nullable tp_observations.sentiment numeric(4,3) (additive; existing RLS/grants cover it). Runner persists it.
  • Project page gains a "How AI talks about you" card (Positive / Neutral / Negative + score). +7 tests.

Improved

  • Leaderboard generator now runs two gentle retry passes (concurrency 3, 45s timeout) over *transient* errors — recovering the ~96 marquee brands (bankofamerica, cloudflare, hsbc…) that timed out under load — while leaving deterministic robots.txt/anti-bot skips alone. data.json regenerated.
  • Email deliverability: the Resend mailer now supports one-click List-Unsubscribe (RFC 8058) headers (Gmail/Yahoo bulk requirement); wired into the domain-expiry reminders. (Note: transactional email was already fully implemented — this hardens inbox placement.)

Config only (no code change)

  • GA4 is already wired (consent-gated, CSP-nonce) — set NEXT_PUBLIC_GA_ID to activate.

Migration

  • Run supabase/0004_observation_sentiment.sql (or supabase/Total_incorporating_0004.sql).

v1.0.0

Added — Increment I: Agency workspaces + seats (the final AIV increment) 🎉

  • Team seats. An Agency (or Growth) owner can invite teammates by e-mail from /account; an invited teammate shares read + day-to-day collaboration (prompts & competitors) on all the owner's tracking projects. Project create/delete and billing stay owner-only.
  • New table tp_members (migration 0003) with its own RLS (owner manages seats; a member reads only rows naming them). The six tracking tables are widened from "owner-only" to "owner OR member" via one security definer set-function tp_member_owners() (self + inviters) — chosen to avoid RLS policy recursion and keep every policy a one-liner. SELECT = owner/member; INSERT/UPDATE/DELETE = per-table (prompts/competitors collaborative; projects/runs/snapshots owner-only writes). Cross-tenant negative-test SQL is inline in the migration.
  • Invites are claimed server-side with the service key (claimInvites binds member_user_id when the invited e-mail signs in) — identity never trusted from the client.
  • Correct ownership limits: a project's prompt/competitor/cadence limits now follow the project owner's plan (entitlementsForOwnerId), so a member on a personal free plan can still work a 400-prompt Agency project; the create-project quota counts only owned projects (countOwnProjects). Seats: Growth 2 · Agency 10 (maxSeats).
  • /account gains a Team panel (seat usage, invite, remove, Active/Invited status) + a "Shared with you" card; /projects labels shared projects and hides Delete on them. +3 tests. Dormant until Supabase Auth is configured — single-user behaviour unchanged.

Migration

  • Run supabase/0003_agency_workspaces.sql (or the consolidated supabase/Total_incorporating_0003.sql) in the Supabase SQL editor.

v0.99.0

Added — Increment K: Monitoring & API plan gating (additive, non-breaking)

  • API keys are now plan-tiered. A signed-in user minting a key gets it on their subscription tier (apiLimits: free 100/day, api_dev 500/day, api_scale 4000/day), baked into the key record (plan, dailyLimit, userId); authorizeAndMeter enforces the key's own daily quota. Anonymous keys keep the free limit unchanged.
  • Monitors are now plan-capped. A signed-in user's monitors belong to them (Monitor.userId) and are limited by plan (monitoringLimits: free 2, mon_pro 20, mon_business 100); the create route rejects over-limit with a 403. Anonymous flow (IP rate-limit) untouched.
  • Multi-product aware: activePlansForEmail returns all active plans, so one user can hold AI-Visibility + Monitoring + API subscriptions at once. Entirely dormant until Supabase Auth is configured — anonymous behaviour is byte-for-byte unchanged. +2 tests.
  • Remaining: I (Agency workspaces + seats — RLS migration + negative tests, next) and J-sentiment.

v0.98.0

Added — tracking Increment J (onboarding: AI-suggested prompts)

  • ✨ Suggest prompts with AI. On a project, one click asks Claude (Haiku, cheap) for 10 realistic buyer questions in the brand's category; tick the ones you want and add them in a batch (src/lib/tracking/suggest.ts, GET /api/v1/projects/[id]/suggest — auth + RLS + rate-limited, addPromptsAction respects the plan's prompt cap). Solves the cold-start — an empty tracking project is now one click from useful. Dormant until ANTHROPIC_API_KEY is set.
  • Remaining: J-sentiment (needs a schema column + is a heuristic — deferred), K (gate Monitoring + API to Stripe entitlements) and I (Agency workspaces + seats) — both are ownership-model + RLS changes, done carefully next.

Changed

  • Crawler User-Agent is now version-stable — Crawlsonar/1.0 (+https://crawlsonar.com/bot; diagnostic scan) (no longer embeds the app version), so verified-bot registrations (Cloudflare Signed Agent) that match the full UA don't drift each release. The performance-check UA is aligned and also points at /bot.

v0.97.0

Added — tracking Increments G+H, and Web Bot Auth directory signing

  • G · Latest-cycle drill-down. Project pages now surface the data we already store per run: "Where AI cites you" (ranked domains the models leaned on — the get-featured list) and "Who wins the answers" (competitors named most this cycle). latestInsights() aggregates the newest cycle's observations.
  • H · Weekly digest email (Pro+). New /api/v1/cron/digest (Mondays 09:00, src/lib/tracking/digest.ts) emails each paid project owner their current share of voice, the week's delta, and top sources/competitors — a retention touch-point. Dormant until Supabase service + Resend are set.
  • Web Bot Auth: the key directory now signs its own response (Signature/Signature-Input, tag http-message-signatures-directory, covering @authority) — Cloudflare requires a signed directory. Request signing was already compliant (Signature-Agent, tag web-bot-auth, JWK-thumbprint keyid, ed25519, @authority+signature-agent).

v0.96.0

Added — good-bot citizenship

  • Public /bot page. Documents the Crawlsonar crawler: its User-Agent, that it reads server-rendered HTML only, respects robots.txt, signs requests (Web Bot Auth), its rate limits, and exactly how to allow or block it (robots.txt snippet + WAF guidance) plus a contact. Linked in the footer + sitemap. This is what Cloudflare/DataDome verified-bot programs and site owners look for.
  • robots.txt enforcement. New src/lib/security/robots-gate.ts (robotsAllows / assertRobotsAllowed, per-host cached, fails open) wired into the automated scanner (scanReadiness — the leaderboard + admin watchlist): we now **skip any site that disallows the Crawlsonar token or * in robots.txt**. +4 tests.
  • User-Agent now points at the bot page — +https://crawlsonar.com/bot — so a site owner who sees it in their logs lands on how to identify/allow/block us.

v0.95.0

Added — leaderboard at scale + AI-Visibility tracking Increment F

  • Leaderboard scaled to ~1000 sites. src/lib/leaderboard/sites.ts expanded from 138 to 970 unique well-known domains across ~25 categories; the batch runner now de-dupes and tolerates bad domains (graceful "error"). The "Ranked by composite readiness" table is now paginated (50/page), sortable by every column, and searchable across every column (src/components/leaderboard-table.tsx).
  • Per-engine trend view. The project trend chart now stores + shows a snapshot per engine plus the "all" aggregate, with an engine selector (groupSnapshotsByEngine, updated TrendChart).
  • AI-visibility change alerts (Pro+). When a cron cycle moves a project's share of voice ≥10 pts — or the brand drops out of AI answers — the owner gets an email (src/lib/tracking/alerts.ts); threshold-gated, dormant until Resend is configured, free tier excluded.
  • White-label PDF report. GET /api/v1/projects/[id]/report renders a per-project AI-visibility PDF (latest metrics, history table, competitors, prompts) via pdfkit; Agency tier gets it white-labelled (Crawlsonar branding dropped). "Download PDF" on the project page.
  • Tests: +3 (alert change logic).

v0.94.0

Added — AI-Visibility tracking, Increment E (multi-engine, Phase 2)

  • Four more engines. src/lib/tracking/engines.ts: ChatGPT (OpenAI gpt-4o-mini), Perplexity (sonar, web-grounded), Gemini (gemini-2.0-flash) and Google AI Overviews (via SerpAPI — a SERP, not an LLM API), alongside Claude. Each is a separate adapter, dormant until its own key is set (OPENAI_API_KEY / PERPLEXITY_API_KEY / GEMINI_API_KEY / SERPAPI_KEY), all using neutral stateless sessions + cheap models.
  • Cron now samples every allowed engine. runner.ts resolves each project owner's plan → allowed engines, intersects with the keys actually configured, and samples each prompt N× per engine; a single flaky engine call no longer sinks the cycle. Writes per-engine trend snapshots + an "all" aggregate. Plan gating updated (Pro = Claude/ChatGPT/Perplexity; Growth+ adds Gemini + Google AIO).
  • Margin: one global daily budget across all engines (TRACKING_DAILY_BUDGET), per-engine cost estimates. Tests: +3 (engine gating); entitlement engine test updated for Phase 2. Env documented.

Changed

  • Copy button next to the header IP — a small clipboard icon beside the desktop nav IP copies it to the clipboard (with a brief ✓).

v0.93.0

Added — AI-Visibility tracking, Increment D (the payoff: trend view)

  • Share-of-voice trend chart on /projects/[id] — pure-SVG, theme-aware two-series chart (share of voice + mention rate, %) over time from tp_trend_snapshots, plus latest-cycle stat tiles and a "12% → 34% since <date>" delta. Empty state until the first cron cycle. src/components/tracking/trend-chart.tsx + listSnapshots.
  • This completes the tracking product loop end to end: create project → add prompts/competitors → scheduled cron samples Claude → watch the trend climb. All dormant until Supabase Auth + ANTHROPIC_API_KEY are configured.

v0.92.0

Added — AI-Visibility tracking, Increment C (the sampling cron)

  • Tracking engine. src/lib/tracking/runner.ts + /api/v1/cron/tracking (daily, CRON_SECRET-auth, origin-gate exempt): for each due project (isDue from cadence + last_run_at) it runs each active prompt N=3× through Claude (stateless = neutral sessions), records every tp_run + tp_observation, writes a daily tp_trend_snapshot (share of voice), and stamps last_run_at. Service-role Supabase client (src/lib/supabase/service.ts) bypasses RLS for the cross-user job.
  • Margin protection. Global daily Claude-call budget in KV (TRACKING_DAILY_BUDGET, default 2000) + per-run caps (25 projects, 100 prompts). Errors captured to the admin Errors tab. Dormant unless ANTHROPIC_API_KEY + Supabase service are set.
  • Tests: +4 (isDue cadence logic).

Changed

  • Crawler User-Agent now carries the real version — Crawlsonar/0.1 → Crawlsonar/${APP_VERSION} (e.g. Crawlsonar/0.92.0 (+https://crawlsonar.com; diagnostic scan)).

v0.91.0

Added — AI-Visibility tracking, Increment B (accounts + gating + projects)

  • End-user accounts (Supabase Auth, magic-link). /account sign-in (passwordless), /auth/callback code exchange, sign-out. Session is scoped to the tracking/account area only (not global middleware) to avoid the getUser-on-every-request perf trap. Dormant until NEXT_PUBLIC_SUPABASE_URL + NEXT_PUBLIC_SUPABASE_ANON_KEY. New deps @supabase/ssr + @supabase/supabase-js.
  • Entitlement gating. currentEntitlements(user) resolves the user's active subscription (planForEmail) → plan → tracking limits; enforced server-side in every Server Action (defense-in-depth on top of the disabled-in-UI controls).
  • Project management UI. /projects (list + create, gated by maxProjects) and /projects/[id] (add/remove prompts and competitors, gated by plan). All mutations are Server Actions using the user-scoped Supabase client, so RLS enforces per-user isolation — the app never trusts a client-supplied user id.
  • Nav: "Account" in the footer. Env documented in .env.example.
  • Next: the sampling cron (runs prompts N× via Claude → stores runs/observations/snapshots) and the trend-chart UI.

v0.90.0

Added — AI-Visibility tracking, Increment A (domain layer + schema)

  • Schema + per-user RLS. supabase/0002_ai_visibility_tracking.sql (+ regenerated Total_incorporating_0002.sql): tp_projects / tp_prompts / tp_competitors / tp_runs / tp_observations / tp_trend_snapshots, each with owner-only RLS (auth.uid() = user_id, transitively for children) so user A can never see user B's data. Indexed for the cron's due-selection.
  • Analysis engine (the "hard part"). src/lib/tracking/analysis.ts — token-boundary brand detection, competitive ranking (brand's position among named brands), cited-source extraction, and share-of-voice aggregation across the N sampling runs. Fully unit-tested.
  • Claude adapter. src/lib/tracking/claude.ts — stateless Haiku calls (= clean, neutral sessions by construction; no account bias), small max_tokens, dormant until ANTHROPIC_API_KEY.
  • Plan entitlements. src/lib/billing/entitlements.ts — plan → tracking limits (projects, prompts, cadence, N runs, competitors, history, engines); engines intersected with LIVE_ENGINES so we never promise an engine the runner can't execute (Phase 1 = Claude only).
  • Tests: +12 (analysis + entitlements). Next increments: Supabase Auth (magic-link) + entitlement gating, the sampling cron, and the project UI + trend chart.

v0.89.1

Changed

  • Admin tables now paginate (10/25/50). The Errors and Users tabs use the reusable DataTable (page-size selector + pagination) instead of raw dumps — honouring the "all data tables paginate" rule. New client tables: errors-table.tsx, subscribers-table.tsx, waitlist-table.tsx.
  • Early-access waitlist is now visible in admin. The Users tab shows a second paginated table of the pricing-page early-access signups (email · wanted plan · when) alongside live subscribers — so the pre-billing list is actionable at launch.
  • /pricing “Why upgrade if the tools are free?” section — condensed objection-handler (track over time · representative data · alerts & reports) with a link to the full AI-Visibility case. Reinforces the value story on the money page.

v0.89.0

Added — Stripe billing (sandbox-ready, dormant until keys)

  • Full subscription checkout. stripe SDK + a billing catalogue (src/lib/billing/plans.ts, single source of truth: 7 plans × monthly/annual, annual = ×10). POST /api/v1/billing/checkout creates a Stripe Checkout Session (subscription mode, 14-day trial, promo codes) resolving the price by lookup_key; the pricing page swaps its waitlist CTA for a real "Start 14-day trial" when NEXT_PUBLIC_BILLING_ENABLED=1, and falls back to the waitlist otherwise.
  • Webhook sync. POST /api/v1/billing/webhook verifies the Stripe signature and upserts a subscriptions store (src/lib/billing/subscriptions.ts, durable KV / in-memory) from checkout.session.completed and customer.subscription.*. Errors captured to the admin Errors tab.
  • Customer portal + result pages. POST /api/v1/billing/portal (opened via the post-checkout session_id, so no email-based billing exposure) + /billing/success (shows plan/trial/receipt + Manage-billing) and /billing/cancel.
  • Admin Users tab is now real — lists live subscribers (email, plan, billing, status, renews) from the store.
  • Setup script: npm run stripe:setup (STRIPE_SECRET_KEY=sk_test_… ) creates all products + prices with lookup_keys, idempotently. Env documented in .env.example.
  • Everything is dormant until STRIPE_SECRET_KEY (+ webhook secret) are set — the app behaves exactly as before without them. Tests: +6 (catalogue math + subscriptions upsert/list).

v0.88.0

Added — /pricing page (pre-Stripe, list-building)

  • New /pricing with a monthly/annual toggle (annual billed for 10 months = 2 months free, pre-selected for higher LTV). Three product blocks — AI Visibility (Free/Pro $34/Growth $119/Agency $299), Monitoring (Free/Pro $12/Business $39), API (Free/Dev $24/Scale $89) — prices trimmed from the research card.
  • Cognitive-bias-driven, honest copy: reciprocity banner ("all 55 tools stay free"), competitor anchor (struck-through $189–199 vs Growth), center-stage "Most popular" featured tier, default-annual toggle framed as a gain, loss-framed objection link. No fake scarcity/countdowns.
  • Early-access capture (pre-billing): paid CTAs open a modal that records the email + wanted plan to a new waitlist store (src/lib/waitlist/store.ts, durable KV / in-memory) via POST /api/v1/waitlist (rate-limited + honeypot). Converts to real Stripe checkout later; builds the launch list now.
  • Nav: "Pricing" added to the desktop header, mobile menu and footer.

v0.87.1

Added

  • AI Visibility: “Can’t I just ask ChatGPT myself?” objection-handler. Customer-facing section on /tools/ai-visibility that answers the #1 GEO objection head-on — non-determinism (one chat = one sample), account contamination (clean neutral sessions), breadth (many prompts × engines), trend-over-time as the real value, competitor + cited-source extraction, and drop-out alerts. Honest (the free check is framed as a baseline snapshot) and written to bridge toward the paid tracking tiers (loss-framed close). Same copy will seed /pricing.

v0.87.0

Added — tabbed admin + error monitoring

  • Admin is now tabbed: Dashboard · Errors · Users. /admin keeps everything that was there under Dashboard (extracted to src/components/admin/dashboard-tab.tsx); tabs switch via ?tab= (src/components/admin/admin-tabs.tsx).
  • Errors tab — real error monitoring. New capped error log (src/lib/errors/store.ts, durable KV in prod / in-memory locally, last 500) with a fail-safe captureError() helper (src/lib/errors/capture.ts). Server failures are now recorded from the reminders cron, monitor cron and Resend email sends; uncaught client-side errors are reported via a global ErrorReporter (window.onerror / unhandledrejection) → POST /api/v1/errors (rate-limited, size-capped). The tab lists newest-first with time / kind / source / message / where, plus a Clear log action (DELETE /api/v1/admin/errors, admin-only).
  • Users tab — placeholder for the accounts/subscribers view we'll build with billing.
  • Tests: +5 (error store record/list/clear/cap, captureError never-throws & non-Error handling).

v0.86.1

Fixed

  • "Remind me before it expires" button was dead. The Turnstile widget mounts on component load via a useEffect([]), but its container only rendered *inside* the save form (shown after a lookup), so it never initialised and no token arrived — leaving the submit button permanently disabled. The Turnstile container now mounts at the component root (interaction-only, so still invisible), so the token is ready when the user submits.
  • Privacy-policy link in the consent notice now opens in a new tab (target="_blank" rel="noopener noreferrer") so it doesn't drop someone out of the form mid-signup.

Changed

  • Expiry reminder offered on every domain check. The /tools/whois-lookup page now includes the reminder opt-in ("Never lose this domain") — for any expiry date, even years out — not just the dedicated tool.
  • Visitor IP moved into the desktop header nav (left of "Tools"), a compact IP · x.x.x.x inline via a new variant="nav" on YourIp. On mobile the full strip stays at the bottom of the homepage (now sm:hidden so it isn't duplicated on desktop).

v0.86.0

Added — Domain Expiry Reminder (free email reminders + list builder)

  • New tool /tools/domain-expiry-reminder. Check any domain's registry expiry (live via RDAP), then opt in to be emailed at 90 / 60 / 30 days before it lapses. Two-step UI (src/components/domain-expiry-reminder.tsx): look up the date first (show value), then capture email with an explicit consent checkbox + Turnstile.
  • Full reminder loop. New src/lib/reminders/{types,store,email}.ts: durable KV store (Supabase/Upstash) with filesystem fallback and a hashed domain+email pair-index for idempotent de-dup; branded Resend email with a working one-click unsubscribe; decideReminder threshold engine (sends only the single most-urgent note when someone joins late, never double-sends).
  • API + cron. POST/GET /api/v1/reminders (authoritative RDAP lookup server-side, rate-limited, Turnstile, consent-gated), POST /api/v1/reminders/unsubscribe + /reminders/unsubscribe confirm page, and a daily cron /api/v1/cron/reminders (added to vercel.json + origin-gate exempt) that re-checks RDAP at each threshold to detect a renewal and skip nagging.
  • Monetisation (honest, light-touch). The reminder email cross-sells our own uptime/SSL monitoring (100% margin) and carries an optional REGISTRAR_AFFILIATE_URL "renew or transfer & save" CTA — dormant until set (renewals are poorly commissioned; transfers/new-regs are what pay). The real asset is the high-intent, recurring email list.
  • Compliance: explicit opt-in consent + timestamp, one-click unsubscribe, sender identity (Northstar Infinity Works Ltd), rate-limit + Turnstile so nobody can sign another address up. Dormant-until-configured for email (EMAIL_*).
  • Tests: +16 (threshold engine incl. late-join & renewal, store create/get/find-by-pair/token-safety, dormant email).

v0.85.0

Added — admin behind Cloudflare Access (Zero Trust)

  • Dedicated admin host. New ADMIN_HOST (e.g. admin.crawlsonar.com): src/proxy.ts serves the dashboard at that host's root (redirects / → /admin) and **refuses /admin* + /api/v1/admin/* with 404 on every other host**, so the marketing domain no longer exposes an admin login. Dormant until ADMIN_HOST is set.
  • Cloudflare Access JWT verification (second layer). New src/lib/security/cf-access.ts verifies the Cf-Access-Jwt-Assertion header — RS256 signature against the team JWKS (cached, refetch-on-kid-miss), plus aud / iss / exp/nbf checks — so a request reaching the origin without passing Access (e.g. a spoofed Host on the raw Vercel URL) is rejected. Dormant until CF_ACCESS_TEAM_DOMAIN + CF_ACCESS_AUD are set.
  • Unified admin gate. src/lib/admin/guard.ts now resolves either mode: with Access configured the signed identity is the sole app-layer auth (the legacy token form is skipped and its cookie ignored); otherwise it falls back to the ADMIN_TOKEN cookie for local/dev. Who may sign in is decided entirely by the Cloudflare Access policy — the app keeps no email/domain allow-list. The admin page shows the signed-in email and, in Access mode, Sign out ends the edge session (/cdn-cgi/access/logout).
  • Tests: +10 (Access JWT verify: valid / forged / bad aud / wrong iss / expired / non-RS256 / disabled; proxy admin-host routing: dormant / refused-off-host / dashboard-on-host / apex-untouched).

v0.84.0

Security (hardening from the full audit)

  • Safe JSON-LD serializer. New src/components/json-ld.tsx (<JsonLd> / serializeJsonLd) escapes <, >, & and U+2028/U+2029 that JSON.stringify leaves raw — closing the theoretical </script> breakout in structured-data blocks. Applied to the site-wide + reused/content-driven spots (layout org data, ToolFaq on every tool page, leaderboard, guide articles). +2 tests. (Inputs are static today; this makes the pattern safe by construction.)
  • Lead email is now format-validated — z.string().trim().toLowerCase().max(254).pipe(z.email()) instead of a bare length cap (rejects junk before storage/send).
  • Admin sign-in rate limit is now durable — switched from the per-instance in-memory limiter to rateLimitKv (holds across serverless instances), on top of the existing Turnstile + constant-time compare.
  • Extra response headers: HSTS gains preload; added Cross-Origin-Opener-Policy: same-origin and X-Permitted-Cross-Domain-Policies: none. (Deliberately NOT Cross-Origin-Resource-Policy — the embeddable AI-Ready badge is loaded cross-origin by design.)
  • Dependencies: pinned postcss to 8.5.16 via overrides (Next bundled a flagged 8.4.31) — npm audit is now 0 vulnerabilities.

v0.83.0

Added

  • Resend email delivery wired. The email stub is now a real send: src/lib/leads/email.ts exposes sendEmail() that POSTs to Resend's API with Authorization: Bearer (accepts EMAIL_API_KEY or RESEND_API_KEY, plus EMAIL_FROM), and report deliveries (sendReportEmail) and monitoring alerts (src/lib/monitor/alerts.ts) now use it with branded HTML. Fails safe (never throws into a route/cron); dormant until EMAIL_API_KEY + EMAIL_FROM are set, logging intent otherwise. New admin Email (Resend) card + /api/v1/admin/email route to check status and send a test email. +5 tests (466 total).

v0.82.1

Added

  • Mobile hamburger menu. Below the sm breakpoint the inline header nav is replaced by a hamburger button (src/components/mobile-nav.tsx) that opens a full-width dropdown with the nav links + Domain audit; the theme toggle stays in the bar. Closes on link tap, on Escape, or by toggling. Desktop nav unchanged.

v0.82.0

Added

  • Light / Dark / Auto theme toggle (geo-aware). A 3-way switch (☀ / ◐ / ☾) in the header. Tailwind's dark: variant now keys off a data-theme attribute (via @custom-variant) instead of the OS media query, so the choice is controllable and persisted in localStorage. A beforeInteractive init script sets the theme before first paint (no flash). Auto resolves via Vercel geo: src/proxy.ts reads x-vercel-ip-timezone on HTML navigations and sets a short-lived geo_theme cookie (day → light, night → dark for the visitor's location); the client falls back to its own clock, then the OS preference. A prefers-color-scheme CSS fallback covers the no-JS case.
  • WebMCP (progressive enhancement). src/components/web-mcp.tsx registers Crawlsonar tools with the emerging navigator.modelContext.provideContext() API so an in-browser AI agent can use the site's actions — list_crawlsonar_tools, check_ai_readiness, check_agent_readiness (calling our public endpoints). Feature-detected, so it's a no-op in every browser that lacks the API today — an early-mover / dogfood flex for an agent-readiness product, with zero downside.

v0.81.0

Changed

  • Homepage rebuilt for laypeople + conversion. Reworked the front page around one clear job, plain language, and ethically-applied cognitive biases (real data only — no dark patterns): — Hero: a friendly question headline ("Can AI find and recommend your website?") + plain-English subhead (60 seconds, 0–100 score, free, no signup), and a single prominent CTA — removed the jargon (llms.txt) and the competing elements (IP strip, tool search) from above the fold. — Trust line anchors on real leaderboard data: "We've scored 138 of the world's best-known sites — the median is 78/100" with a "See where you'd land" curiosity hook. — "What you'll get" (score / issues / fixes) kills ambiguity before the click; "How it works" is three tiny steps; a benchmark section reuses the leaderboard (median 78, 15% blocked) for social proof + relativity; "What we check for you" reframes tools as outcomes a non-expert cares about. — Secondary paths (full audit, all 50+ tools, tool search, the IP strip) moved below the fold so they never compete with the primary action. Verified in light/dark, desktop/mobile.

v0.80.0

Added

  • Content Signals (Cloudflare Content Signals Policy) — end to end. — Published on our own robots.txt. Converted the robots.txt from Next's structured MetadataRoute to a text route handler (src/app/robots.txt/route.ts) so it can emit custom directives, and added Content-Signal: search=yes, ai-input=yes, ai-train=yes (we're a GEO product — all uses allowed) to both the * group and the AI-crawler group. — Detected in the Agent Readiness Checker. New src/lib/checks/content-signals.ts parses a Content-Signal: directive; the checker adds a Content Signals declared in robots.txt check (weight 8) with an info finding + example when absent. Weights rebalanced (AI-crawler-rules 12→8, OpenAPI 12→8) so the module still sums to 100. +3 tests. — Generated by the AI robots.txt Generator. The generated policy now includes a Content-Signal directive derived from your allow/block choices (search / ai-input / ai-train), e.g. allow-all → all yes, block-all → all no, block-training → ai-train=no. +4 tests (461 total).

v0.79.0

Added

  • mysentry uptime heartbeat. Crawlsonar now sends a heartbeat to mysentry (the operator's monitoring app) so its panel shows Crawlsonar alive — a dead-man's switch: if the pings stop, mysentry alerts. New src/lib/mysentry.ts POSTs { monitor, ok, processed } with Authorization: Bearer <token> to mysentry's /api/ingest/heartbeat (fire-and-forget, never throws into the cron; +5 tests). Configured from a new admin card — paste your mysentry project ingest token (stored server-side in KV, never returned to the client), pick a monitor name, and Send test heartbeat to confirm it lands. The hourly monitor cron carries the heartbeat (ok reflects whether the run had errors). Admin APIs /api/v1/admin/mysentry (+ /test) are session-guarded. Dormant until a token is set.
  • /api/health — a lightweight public liveness endpoint (GET/HEAD → 200 with { status, version, store, time }), handy for any external HTTP uptime check.

v0.78.0

Added

  • DNS for AI Discovery (DNS-AID) check in the Agent Readiness Checker. New src/lib/checks/dns-aid.ts probes _index._agents.<domain> and _a2a._agents.<domain> over DoH for ServiceMode SVCB/HTTPS records (RFC 9460) and reports whether the discovery answer is DNSSEC-authenticated (AD flag). Surfaces a concrete finding + fix (publish ServiceMode SVCB/HTTPS with alpn+endpoint, DNSSEC-sign the zone) with an example record, plus a dedicated explainer on /tools/agent-readiness linking the IETF DNS-AID draft and RFC 9460. Wired into the report (weight 10; llms.txt & MCP trimmed 15→10 to keep the module total at 100). +3 tests.
  • Leaderboard average & median. The leaderboard now shows median + average of composite, AI-readiness and security across the scored sites (computeStats in src/lib/leaderboard/score.ts, +2 tests). When a visitor runs the AI Readiness Checker, a new "How you compare" panel shows their score against the leaderboard's AI-readiness median & average (of the well-known sites), with a distribution bar — stats passed server-side so the client bundle stays lean.
  • Turnstile on admin sign-in. The admin login form now carries an (invisible) Turnstile check and the login route verifies it — defense-in-depth on top of the existing 5/5-min brute-force rate limit. Dormant unless Turnstile is configured.

Changed

  • robots.txt: AI crawlers now get unrestricted access. The AI-crawler group (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, …) is now Allow: / with no disallows — maximally readable and citable by AI, which is the point of a GEO product. The general * group keeps /api/, /r/, /admin, /monitor/ out.

v0.77.0

Added

  • Admin domain watchlist + reusable data table. New reusable DataTable (src/components/admin/data-table.tsx): page-size selector (10/25/50), pagination, and optional mass-selection with bulk actions — applied across the admin. New Watched domains section in /admin: add domains, scan now (live AI-readiness + security composite via the shared scanReadiness engine), and a full lifecycle — Active (Archive / Bin), Archived (Unarchive / Bin), Bin (Restore / Delete now). Binned domains auto-delete after 30 days (lazy on read + in the daily cron). crawlsonar.com and mysentry.co.uk are seeded so the operator's own properties are tracked out of the box. Durable store in KV with an in-memory fallback for local dev; state-machine, purge, normalisation and dedupe are pure and unit-tested (+7 tests, 443 total). Admin APIs (/api/v1/admin/domains + /action + /settings) are session-guarded.
  • Daily domain auto-scan cron. New /api/v1/cron/domains-scan (Vercel Cron, 0 6 * * *) purges expired binned domains then scans the day's least-recently-checked auto-scan domains, up to the operator-set auto-scan/day count (a box in the admin). Added to the origin-gate exemption in src/proxy.ts (Vercel Cron bypasses Cloudflare, so it must not be 403'd by origin cloaking) — covered by a proxy test.
  • The existing Recent tests and Recent leads admin tables are now paginated via the same DataTable.
  • scanReadiness extracted to src/lib/leaderboard/scan.ts and shared by the leaderboard generator and the watchlist (DRY).

v0.76.0

Added

  • The AI & Web Readiness Leaderboard (/leaderboard). We scan a curated list of well-known public global sites through our own engine and rank them by a composite of AI readiness (55%) + HTTP security & headers (45%) — a data-journalism / backlink-magnet page (only big, public entities; no shaming of small sites). Sites that answer with an anti-bot challenge are detected via the bot-wall engine, marked blocked and surfaced separately with the vendor — a telling stat in its own right ("N% blocked our fetch — and, likely, AI crawlers too"). Each row links into the AI Readiness Checker (funnel). Static snapshot in src/lib/leaderboard/data.json, generated by npm run gen:leaderboard (scripts/gen-leaderboard.ts); the page prerenders from it, so zero per-request scanning. Pure composite/ranking logic in src/lib/leaderboard/score.ts (+5 tests, 436 total). First scan: 138 sites, 113 scored, 21 (15%) blocked. Linked from the header + footer nav.

v0.75.0

Added

  • Blacklist Check tool (/tools/blacklist-check) — "is your IP or domain on a spam blocklist?". New src/lib/checks/blacklist.ts engine queries DNSBLs over DoH: an IP is checked against IP lists; a domain is checked against domain lists (DBL) and its MX servers' IPs (up to 3) against the IP lists. Results feed the standard explainable score (a listing is a high-severity issue that caps the score; unreachable lists are excluded, never counted as clean). Per-listing results cached in KV for ~1h. New API route /api/v1/tools/blacklist (zod + durable per-IP rate limit 15/h + Turnstile-gated) and the tool is registered in the search index.
  • Dormant Spamhaus DQS integration. With SPAMHAUS_DQS_KEY set, the tool queries Spamhaus ZEN (IP) + DBL (domain) via the Data Query Service — the sanctioned, serverless-friendly path (Spamhaus blocks public-resolver queries and forbids commercial use of the free mirror). Without a key it runs the best-effort public list (SpamCop) and shows a "limited coverage — connect a free Spamhaus DQS key" notice. Blocked/misconfig sentinels (127.255.255.x) are treated as *unavailable*, never a false listing. +8 tests (431 total).

v0.74.1

Fixed (hardening, dormant-safe)

  • Origin gate no longer breaks the service webhooks when enabled. The origin-cloaking gate (src/proxy.ts, dormant until ORIGIN_SHARED_SECRET is set) now exempts two self-authenticating service endpoints that legitimately arrive without the Cloudflare-injected header: /api/v1/inbox/ingest (Cloudflare Email Worker subrequest — a Worker's fetch into our zone isn't guaranteed to pick up the Transform Rule) and /api/v1/cron/monitor (Vercel Cron invokes the deployment directly, never traversing Cloudflare). Both stay protected by their own strong, constant-time secrets (INBOX_INGEST_SECRET, CRON_SECRET), so exempting them doesn't weaken cloaking. Exact-match allow-list (never prefix). Prevents inbound test emails and monitoring from silently dying the moment cloaking is switched on at launch. +6 tests (423 total) covering dormant pass-through, gated 403, correct-header pass, and both exemptions.

v0.74.0

Changed

  • Turnstile is now invisible (appearance: "interaction-only"). The widget no longer renders a visible box in the common case — it stays zero-height and still returns a token, only becoming visible if Cloudflare genuinely needs an interactive challenge. Fixes the bulky/ugly widget on the inbox and AI Visibility tools.

Added / Fixed (privacy policy)

  • Cloudflare Turnstile privacy addendum. New "Bot protection (Cloudflare Turnstile)" section on /privacy explaining what Cloudflare processes (IP + browser/device/interaction signals), that it's strictly-necessary and not used for cross-site tracking/ads, lawful basis (legitimate interests), and a link to Cloudflare's Turnstile Privacy Policy. Turnstile added to the processor list.
  • Corrected the GA4 contradiction. The privacy policy still claimed "we set no cookies… no cookie-consent banner is required", which became false when GA4 + the consent banner shipped (0.71–0.72). The Cookies section is rewritten into strictly-necessary vs consent-based analytics; the Summary and Usage-analytics sections now describe Google Analytics 4 under Consent Mode v2 (disabled until you accept), with Google added to the processor list. Last-updated bumped to 10 July 2026.

v0.73.1

Changed

  • Compact "Your IP" strip. The IP box (homepage + What Is My IP tool) was a tall card with a large text-lg address; it's now a single-line pill (px-3 py-2, text-sm) showing IP · version · browser/OS · Copy on one row, taking far less vertical space.

v0.73.0

Added

  • Cloudflare Turnstile bot mitigation (no-CAPTCHA) on the two costly/abusable endpoints: inbox address minting (/api/v1/inbox/new, creates DB rows) and AI Visibility (/api/v1/tools/ai-visibility, spends LLM tokens). Defense-in-depth — the per-IP rate limits stay as the cost floor; Turnstile raises the bar for automation on top. New src/lib/security/turnstile.ts verifies the token server-side with Cloudflare's siteverify (fails closed on a missing/denied token, open if Cloudflare itself is unreachable — same philosophy as the rate limiter, and the per-IP limit still caps abuse in that window). New src/components/turnstile.tsx useTurnstile() hook loads the widget script once and hands the client a single-use token sent in the x-turnstile-token header. Dormant until configured: with no NEXT_PUBLIC_TURNSTILE_SITE_KEY + TURNSTILE_SECRET_KEY nothing renders and no token is required, so behaviour is unchanged — requiring both keys avoids locking users out with a half-config. CSP widens to https://challenges.cloudflare.com (script-src + frame-src) only when a site key is set; dormant deployments keep the tighter frame-src 'none'. +8 tests (417 total).

v0.72.0

Added

  • GDPR cookie consent + Google Consent Mode v2. GA4 now loads in a consent-aware state with all storage denied by default (analytics_storage, ad_storage, ad_user_data, ad_personalization set to denied, with wait_for_update: 500), set in the GA init before gtag('config') — so no analytics cookie is written until the visitor opts in. A new bottom cookie banner (CookieBanner, production-only) offers Accept / Reject with a link to the privacy policy; on Accept it calls gtag('consent','update',{ analytics_storage:'granted' }) and persists the choice to localStorage (cs_consent). Returning visitors who accepted get consent re-applied immediately on load; those who rejected (or never chose) stay denied. A "Cookie settings" link in the footer reopens the banner so the choice can be changed at any time (GDPR withdrawal of consent). Ad-related signals stay denied regardless — this is analytics-only.

v0.71.1

Fixed

  • Inbox test: friendlier rate-limit message. When the durable limiter on /api/v1/inbox/new returns 429 (a visitor minted many test addresses quickly), the client now shows a dedicated "You've created a lot of test addresses just now. Please wait a minute, then refresh this page." notice instead of the misleading generic "Couldn't start the test" error.

v0.71.0

Added

  • Google Analytics 4 (property G-FG8G8SKBVB) loaded via next/script (afterInteractive) in the root layout — production only, so local dev doesn't pollute the property. Configurable via NEXT_PUBLIC_GA_ID. The Content-Security-Policy was widened to allow it: script-src adds www.googletagmanager.com; connect-src adds the Google Analytics collection domains. (Note: GA4 sets cookies — a GDPR cookie-consent gate is the recommended next step.)

v0.70.0

Changed

  • Hardening: durable, shared rate limiting on the abusable/costly endpoints. Added rateLimitKv — a fixed-window limiter backed by the KV store (Supabase), so the limit holds across all serverless instances, unlike the in-memory limiter whose counter is per-instance (weak on Vercel). Applied to key-minting (/api/v1/keys), AI Visibility (/api/v1/tools/ai-visibility, each call spends LLM tokens), shareable-report creation (/api/v1/reports, re-runs a scan), and inbox-test address minting (/api/v1/inbox/new, creates DB rows). Falls back to the in-memory limiter when KV is unconfigured and fails open on a KV error (never blocks a legitimate user). +2 tests (409 total).

v0.69.0

Added

  • Charts in the admin dashboard. The plain tables are now visual: an Activity — last 14 days bar chart (total events per day, with the tests portion highlighted), and horizontal bar charts for the Funnel and Tests by tool. Rendered server-side as pure CSS/SVG bars — no chart-library dependency, theme-aware, with hover tooltips. Backed by a new daily-activity aggregation in admin-data.ts.

v0.68.0

Changed

  • Inbox test: a clear 3-step "how to use" on the tool (copy the address → email it → read your score), so a first-time visitor knows exactly what to do.
  • Inbox tests now show in the admin "Recent tests" log authoritatively — logged server-side in the ingest endpoint the moment an email is analysed (tool inbox-test, score, sender as the target), so they appear even if the visitor closed the tab. Removed the duplicate client-side event.

v0.67.0

Added

  • Inbox Deliverability Test (/tools/inbox-test) — the "send us an email, get a spam/inbox score" tool (mail-tester style). The visitor gets a unique one-time address (test-<token>@crawlsonar.com); Cloudflare Email Routing forwards the raw message to POST /api/v1/inbox/ingest (authenticated by a shared INBOX_INGEST_SECRET, timing-safe), which parses the receiving MTA's Authentication-Results (SPF/DKIM/DMARC verdicts), extracts the sending IP, resolves reverse DNS, checks a couple of bulk-sender signals, and stores a scored report in Supabase keyed by the token. The page mints the address and auto-polls until the email lands, then shows the inbox-readiness score with fixes and Learn links. — New endpoints: POST /api/v1/inbox/new, POST /api/v1/inbox/ingest, GET /api/v1/inbox/result/[token]. Pure, unit-tested MIME-header parsing + scoring (inbox.ts, 7 tests, 407 total). Hex token survives MTA lowercasing. Dormant-until-configured (needs Supabase + INBOX_INGEST_SECRET). First revenue-shaped tool (free now; credit-gating to follow with the paid OG tier).

v0.66.0

Changed

  • Full target coverage in the admin "Recent tests" log. The target domain/URL is now recorded for every domain/URL tool (18 clients: DNS, SSL, HTTP headers, robots, sitemap, performance, link-preview, security scanner, structured data, CT subdomains, redirect, DNS focus, domain audit, plus the shared ReportScanClient covering SEO/WHOIS/deliverability/etc., and the earlier HSTS/DNSSEC/DKIM/Agent Readiness). Tools without a single domain target by nature (secret/hex generator, user-agent parser, log analyzer, email-header paste) correctly show none. So the admin audit log now answers "who tested what & when" with the actual target.

v0.65.0

Added

  • Web Bot Auth — signed outbound crawler requests (RFC 9421 HTTP Message Signatures). When a signing key is configured, safeFetch now signs every hop of our outbound requests with an Ed25519 signature (Signature / Signature-Input / Signature-Agent headers, tag="web-bot-auth"), so the sites we scan can cryptographically verify the request genuinely came from Crawlsonar rather than an impersonator. The matching public key is published at /.well-known/http-message-signatures-directory (JWKS). — Dormant-until-configured: set WEB_BOT_AUTH_PRIVATE_JWK (an Ed25519 private JWK, server-only secret) to enable; with no key we don't sign and the directory 404s — zero overhead and safe to ship. — Pure, unit-tested core (web-bot-auth.ts): RFC 9421 signature-base construction, RFC 7638 JWK thumbprint key id, and a full sign→verify round-trip test (generate an Ed25519 key, sign a request, verify the signature against the published public key). 5 tests (400 total). Completes the honestly-achievable agent-readiness set.

v0.64.0

Added

  • Admin: "Recent tests — who, what & when" audit log. The admin dashboard now shows a live table of individual test runs across every tool: when (timestamp), what (tool + target domain), who and the score. Captured server-side on the events pipeline, so it covers all 24 tools automatically. — Privacy-respecting "who" (new who.ts, unit-tested): we store the two-letter country (from the CDN geo header) and a short salted one-way hash of the IP (cid) — never a raw IP or user agent, keeping the cookieless/no-PII discipline while still letting the owner tell repeat visitors and abuse apart. — Target domain is recorded for the scans that report it (HSTS, DNSSEC, DKIM, Agent Readiness now; more to follow); who/when/tool is captured for every tool. Durable on Supabase.

v0.63.0

Added

  • Markdown content-negotiation — an AI agent that requests the homepage with Accept: text/markdown now gets a clean, link-rich markdown briefing (generated from the tool registry at /api/markdown) instead of the full HTML. Wired into the existing origin proxy, scoped to / and gated on the Accept header, so ordinary browsers (which send text/html) are completely unaffected. Completes another agent-readiness signal — the last of the honestly-achievable ones before Web Bot Auth.

v0.62.0

Added

  • Agent Skills discovery index at /.well-known/agent-skills/index.json (Agent Skills Discovery RFC v0.2.0). Generated live from the public API tool registry: every check is a "skill" an agent can invoke via POST /api/v1/run, each with a genuine sha256 digest of its descriptor (not a placeholder) and clear invocation instructions (HTTP + MCP). Part of making crawlsonar itself agent-ready.

v0.61.0

Added

  • Supabase (Postgres) as the durable-store backend — src/lib/kv.ts now speaks to two interchangeable backends behind one interface, chosen by env, with Supabase preferred: set SUPABASE_URL + SUPABASE_SECRET_KEY (the new sb_secret_… key; legacy SUPABASE_SERVICE_ROLE_KEY also accepted) and every dormant feature (shareable reports, benchmark histogram, monitoring, API keys, AI cache) lights up. Upstash/Vercel KV (Redis REST) remains as the alternative; with neither set, the app falls back to the filesystem stores (local dev unchanged). — Redis-style ops map onto three tables + atomic RPC functions: kv_store (SET/GET/INCR + TTL, with expiry-aware counter reset), kv_hash (HINCRBY/HGETALL), kv_list (RPUSH/LTRIM/LRANGE, negative-index aware). SQL in supabase/0001_kv_store.sql (+ consolidated Total_incorporating_0001.sql): RLS enabled on every table with no policies (only the secret key / service_role bypasses), and RPC EXECUTE revoked from anon/authenticated — least privilege by default. — The Upstash wire format is byte-for-byte unchanged; +8 unit tests cover the new Supabase path and backend precedence (390 total). Dormant-safe: shipping this changes nothing until the Supabase env vars are set.

v0.60.0

Added

  • MCP Server Card at the SEP path — crawlsonar now also serves its MCP discovery manifest at /.well-known/mcp/server-card.json (SEP-1649 shape: serverInfo, transport, capabilities), in addition to /.well-known/mcp.json, so agents and audits that look for either convention find it. The Agent Readiness Checker now probes both paths too.

v0.59.0

Added

  • Random Hex & Secret Generator (/tools/secret-generator) — generate a cryptographically secure random secret with a user-set length (1–512 bytes) in hex, base64url or base64, with an optional uppercase toggle. Uses the Web Crypto API (crypto.getRandomValues) entirely client-side — nothing is sent to a server. Shows entropy (bits) and character count; one-click copy and regenerate. Live-verified: 32 bytes → 64 hex chars, length control works. Wired into homepage search, sitemap, /tools and llms.txt.

v0.58.0

Added

  • Dogfooding: made Crawlsonar itself agent-ready — added the real, standards-based discovery surface our own Agent Readiness Checker probes (no faking: OAuth is intentionally omitted because we use API-key auth, not OAuth): — /.well-known/security.txt (RFC 9116) — machine-readable security contact. — /.well-known/mcp.json — a manifest describing our live MCP endpoint (/api/mcp, streamable-http, protocol 2025-06-18). — /openapi.json — a compact OpenAPI 3.1 description of the public API (/api/v1/run, /api/v1/keys, /api/mcp). — /.well-known/api-catalog (RFC 9727 linkset) pointing at the OpenAPI + docs. — Discovery Link headers on the homepage response (rel="describedby" → llms.txt, rel="service-desc" → OpenAPI, rel="api-catalog"), emitted from next.config headers. — Together with the existing llms.txt and explicit AI-crawler rules, this takes crawlsonar.com's own Agent Readiness score from failing to a strong pass — the tool now practises what it preaches.

v0.57.0

Added

  • OG Image Generator (/tools/og-image-generator) — a free tool that renders a crisp 1200×630 social-share PNG from a title, subtitle, brand and one of six themes, server-side via next/og (/api/og-image). Instant, no sign-up, no watermark, zero marginal cost. Live preview + one-click PNG download. Live-verified: the endpoint returns a valid 1200×630 image/png. Wired into homepage search, sitemap, /tools and llms.txt. — Sets up the paid tier: the page carries a dormant "AI artwork — coming soon" upsell. The paid path (AI-generated backgrounds via OpenAI gpt-image-1, sold as one-off credit packs through Stripe with a KV credit ledger) is deferred until KV + Stripe + OPENAI_API_KEY are configured — money code shouldn't ship untested against real keys.

v0.56.0

Added

  • "Learn" — a plain-English knowledge base (/learn). Every concept behind our checks, explained twice on its own SEO page: once for a total beginner (with an everyday analogy), once for a professional, plus how-to-fix steps, an FAQ and links to the relevant tools. 13 launch entries covering HSTS, SPF, DKIM, DMARC, DNSSEC, HTTP security headers, structured data, canonical URLs, meta descriptions, AI crawlers & robots.txt, llms.txt, crawler/WAF bot-blocking and SSL/TLS. Each page emits DefinedTerm + FAQPage + BreadcrumbList JSON-LD; the index emits DefinedTermSet. Wired into the header nav, sitemap and llms.txt.
  • "Learn what it means, in plain English →" links on every issue. The shared IssueList now maps each finding to its knowledge-base entry (learnSlugForIssue), so any issue we can explain automatically gets a beginner-friendly Learn link — across every tool at once. For laypeople building their first site, not just pros.
  • New data-driven kb.ts (entries + issue→slug resolver, 6 unit tests, 382 total). Easily extensible — adding a topic is pure content.

Fixed

  • Shareable report links no longer mint dead URLs when the report database isn't connected. On Vercel's per-invocation serverless filesystem (no KV), a created share link would 404 from another instance. The share API now detects that persistence isn't durable and returns a clear "shareable links are being switched on" message instead of a broken link; the Share button shows it gracefully. Activates fully the moment Upstash KV is configured. (Local dev still works via the filesystem store.)

v0.55.0

Added

  • Agent Readiness Checker (/tools/agent-readiness) — a new standalone tool for the emerging "agentic web": where the AI Readiness Checker scores how an LLM reads a *page*, this probes the origin-level discovery surface autonomous AI agents use. Nine real, observable signals: llms.txt, explicit AI-crawler rules in robots.txt, Markdown content-negotiation (Accept: text/markdown), an MCP manifest (/.well-known/mcp.json or /mcp), OpenAPI / API-catalog description, OAuth discovery metadata (/.well-known/oauth-authorization-server / -protected-resource), discovery Link headers, and security.txt (RFC 9116). We only score signals we can actually observe over HTTP/DNS — speculative agentic-commerce / bot-auth proposals are deliberately left out so the score stays honest. — Pure buildAgentReadinessReport + SSRF-safe checkAgentReadiness (parallel probes, never throws). 5 unit tests (376 total). Wired into the homepage search, public API + MCP (agent-readiness), sitemap, llms.txt and /tools. Live-verified: github.com correctly detects its security.txt; crawlsonar.com detects llms.txt + 9 AI-crawler rules.

v0.54.0

Fixed

  • False negatives when a site is behind a bot wall / WAF, and the "dates/privacy" checks. Triggered by a real report on www.ukimmigration.law (privacy/terms and publication dates flagged missing when they exist): — Bot-wall detection (new bot-wall.ts, unit-tested) — when a site sits behind Cloudflare Bot Fight Mode / a managed challenge (often served as HTTP 200 with an interstitial) or another WAF (Akamai, Imperva, PerimeterX, DataDome, Sucuri), our crawler was parsing the challenge page and reporting everything as missing. The AI Readiness Checker now detects the block, marks the unreadable content checks not-applicable instead of failed, and raises one headline finding: *"Your site is blocking automated crawlers"* — with the key insight that the same block hits GPTBot / ClaudeBot / PerplexityBot / Google-Extended / Googlebot, so the content can be invisible to AI and search. Fix guidance points at Cloudflare "Verified bots" / WAF allow rules. — Privacy/terms detection broadened — now matches the link text as well as the href, across privacy / terms / cookie(s) / GDPR / legal / disclaimer / data-protection / impressum. (Fixed the ukimmigration.law false negative.) — Dates detection broadened — the check is about dates being *visible*, so it now also accepts a clearly-visible date in the text ("8 July 2026", "July 8, 2026", "2026-07-08"), plus article:published_time / article:modified_time / itemprop date meta, not only <time> and JSON-LD. — Live-verified on www.ukimmigration.law: privacy + dates now pass (100/A); +7 tests (371 total).

v0.53.0

Added

  • Plain-language layer + "Fix-it" everywhere — for millions, not geeks. Two cross-cutting UX changes to the shared report components, so they land on *every* tool at once (the SEO checker, security scanner, HSTS, DNSSEC, audit — anything using ScoreCard/IssueList): — Plain-English verdict under every score: one honest, non-alarmist line that tells a non-technical visitor what the number means (e.g. *"In plain English: this needs attention — several important things are failing below."* vs *"…this looks healthy — no urgent problems found."*). — One-click fixes — issue Fix text now renders backtick snippets as real inline code, and any fix that contains a copy-pasteable snippet gets a "Copy fix" button (plus a Copy button on example blocks). One consistent affordance across the whole site via a new shared <CopyButton>. — New pure, unit-tested report-format.ts (plainVerdict, splitInlineCode, firstCodeSnippet; 7 tests). Live-verified: HSTS on example.com → Grade F, plain verdict shown, "Copy fix" button on the missing-HSTS fix.

v0.52.0

Added

  • Score benchmarking — the first KV growth loop (moat + linkbait). Every completed scan now contributes its 0–100 score to a per-tool histogram in KV, and the report shows the visitor where they land: *"Better than X% of sites we've checked with this tool · based on N scans."* The distribution improves with every scan (a data moat that can't be copied) and a top-decile result is eminently shareable. — Privacy by design: we store only a tool id and an integer score bucket (bench:v1:<tool>, fields "0".."100") — never the domain, IP or any identifier. Nothing to leak. — Dormant-until-configured: with no Upstash/KV set, the endpoint returns { enabled: false } and the badge renders nothing — so it's invisible and harmless until KV is wired (activates the moment KV_REST_API_URL/KV_REST_API_TOKEN are set). Percentile is also hidden below a 30-scan minimum so it's never misleading. — New pure computePercentile() (unit-tested, 7 tests), recordAndRank()/getRank() (never throw), POST /api/v1/benchmark, and a reusable <BenchmarkBadge> wired into the Domain Audit, HSTS Checker and DNSSEC Checker. Added kvHincrby/kvHgetall to the KV client.

v0.51.0

Added

  • Structured-data (schema.org) presence check in the On-Page SEO Checker. The SEO checker now flags whether a page has any structured data — JSON-LD, microdata (itemscope/itemtype) or RDFa (typeof) — so a missing-schema page is caught in the ordinary SEO audit, with the detected @types shown as evidence and a fix that links to the JSON-LD Generator and Structured Data Checker. Also flags invalid JSON-LD blocks. Live-verified: crawlsonar.com → passed (Organization, WebSite); example.com → failed (no structured data). Reuses the already-tested parseJsonLd; feeds the /tools/seo-checker tool and the domain audit.

v0.50.0

Added

  • HSTS Checker (/tools/hsts-checker) — a dedicated Strict-Transport-Security tool. Fetches the site over HTTPS, parses max-age / includeSubDomains / preload, and queries the official hstspreload.org status API to report whether the domain is actually on the browser preload list. Catches the real footguns: max-age=0 (HSTS disabled), a short lifetime, missing includeSubDomains, a preload directive that isn't eligible, "eligible but not submitted", and — importantly — "on the preload list but the live header no longer qualifies". Pure parseHsts + buildHstsReport, unit-tested; live-verified github.com = 100/A, on the preload list.
  • DNSSEC Checker (/tools/dnssec-checker) — validates the full chain of trust (root → TLD → zone) over DNS-over-HTTPS. Shows the three links (zone signed via DNSKEY, anchored via DS in the parent, validates to root via the resolver AD flag) and flags the dangerous states that quietly take domains offline: DS with no DNSKEY (a live resolution outage — critical), bogus (DS/DNSKEY mismatch after a key rollover — high), island of security (signed but not anchored — medium), and deprecated crypto (RSASHA1 / SHA-1 DS). Pure parsers + buildDnssecReport, unit-tested; live-verified cloudflare.com = 100/A.
  • Both tools are wired into the homepage search, the public API + MCP (hsts, dnssec tool ids), sitemap.xml, llms.txt and the /tools index.
  • 27 new unit tests (HSTS 13, DNSSEC 14). 349 total.

Fixed

  • DNSSEC algorithm parsing — Cloudflare's DoH renders the signing algorithm as a mnemonic name (e.g. 256 3 ECDSAP256SHA256 …), not a number. The first cut parsed it as a number, dropped every DNSKEY, and falsely reported healthy domains (cloudflare.com!) as a "critical broken chain". algNumber() now accepts both the mnemonic and numeric forms; caught in live verification before shipping, with a regression test added.

v0.49.0

Changed

  • Rich, option-by-option educational content on every generator — expanded the below-the-fold SEO/GEO narrative on all 9 generator tool pages so each one explains *what the thing is*, documents *every option*, and warns of the common mistakes. This deepens topical authority (each page now targets the long-tail "what does X option do" queries) and helps users configure correctly the first time: — Security Headers Generator — HSTS (max-age / includeSubDomains / preload), CSP presets (strict / basic / none) + report-only rollout + nonce path, X-Frame-Options (DENY / SAMEORIGIN / none), nosniff, Referrer-Policy, Permissions-Policy. — SPF Generator — a, mx, ip4/ip6, include, and -all/~all/?all/+all, plus the one-record and 10-lookup rules. — DMARC Generator — p, sp, pct, rua/ruf, adkim/aspf, and the safe three-stage rollout. — JSON-LD (Schema) Generator — when to use Organization / WebSite / Article / LocalBusiness / Product, and the visible-content rule. — Open Graph Generator — every og:/twitter: field, image sizing (1200×630), and cache/absolute-URL pitfalls. — CAA Generator — CA issue list, issuewild, and iodef reporting. — MTA-STS & TLS-RPT Generator — mode (testing / enforce / none), MX hosts, max_age, TLS-RPT rua, and how to publish the policy. — AI robots.txt Generator — the three presets (protect-training / allow-all / block-all) and the per-crawler allow/block toggles + opt-out tokens. — llms.txt Generator — what llms.txt is and how the draft is built.
  • Content-only change (no logic touched): build compiles, all 323 tests pass, generator pages remain statically pre-rendered.

Fixed

  • Footer version showed "· v" with no number. Next 16 does not inline env: { NEXT_PUBLIC_APP_VERSION } into the statically prerendered server-component layout, so the value was empty in production. Replaced the env/process.env indirection with a deterministic generated literal (src/lib/app-version.ts), imported by the layout — works identically across static prerender, dynamic render and Vercel serverless. Removed the now-dead fs read + env block from next.config.ts. Verified · v0.49.0 live.

v0.48.0

Added

  • Homepage tool search — a type-to-find search box above "Popular tools" that narrows suggestions as you type across all 40+ tools, with keyword matching (type dkim, hsts, spf, core web vitals…), keyboard navigation and Enter-to-open. Backed by a new canonical src/lib/tools-index.ts (single searchable index with keywords). Makes the growing toolset discoverable.
  • DKIM Checker (/tools/dkim-checker) — the missing third of the SPF/DKIM/DMARC trio as a standalone tool. Looks up a domain's DKIM public key by selector (or probes the common provider selectors), validates it, and reports key type, estimated strength (flags weak ~1024-bit RSA), and testing/revoked flags — with fixes. Pure record parser + report builder, unit-tested. Live-verified against github.com (selector google, RSA, strong). — 11 new unit tests (DKIM parsing/scoring + tool-search ranking). 323 total. Build compiles. HSTS was already covered by the HTTP Header Checker + Security Headers Generator (now instantly findable via search).

v0.47.0

Added

  • Fix Pack (/tools/fix-pack) — the "don't just find problems, fix them" tool. Enter a domain and get copy-paste-ready, pre-filled snippets for the most common SEO/AI/security gaps: HTTP security headers, an AI-crawler robots.txt policy, SPF + DMARC anti-spoofing records, Organization JSON-LD and an llms.txt skeleton — each with where-to-apply guidance and a copy button. Fully client-side (reuses the pure generator libs), so it's instant and private. Live-verified: 6 blocks generate pre-filled for the domain. Registered in /tools, sitemap, llms.txt.

v0.46.0

Added

  • MCP server — Crawlsonar is now agent-native. A zero-dependency Model Context Protocol server at POST /api/mcp (Streamable HTTP, JSON-RPC 2.0) exposes all 13 registry tools (domain-audit, dns, ssl, http-headers, whois, email-deliverability, ai-readiness, llms-txt, performance, core-web-vitals, security, subdomains, link-preview) as agent-callable tools — so an AI assistant can audit a domain itself. Reuses the shared tool registry, so MCP, the public API and the UI run one implementation. initialize / tools/list / tools/call / ping / notifications handled; SSRF-safe; rate-limited. Documented on /api-docs and in llms.txt. — 7 new unit tests over the pure JSON-RPC router. 312 total. Live-verified: full handshake + a real tools/call (whois google.com → registrar + score via MCP).

v0.45.4

Changed

  • Explicit AI-crawler policy in robots.txt. Added a dedicated rule allowing every known AI crawler (GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot, CCBot, Google-Extended and more) — we *want* to be readable and citable by AI (GEO), so we opt in explicitly rather than relying on the wildcard. Also added the host directive and kept /monitor/ out of crawling. Resolves the "no explicit AI crawler policy" finding from the self-audit and lifts the AI Readiness + robots.txt module scores.

v0.45.3

Changed

  • Persuasion-optimized default OG image. Redesigned the social-share card for click-through using honest cognitive-bias techniques: a curiosity gap (a big "?" as "your AI score" — the number you don't yet know), loss framing ("invisible to AI?" in alarm-amber beats "readable by AI"), concrete numbers ("40+ free checks"), recognisable authority (ChatGPT, Claude, Perplexity) and a "Free" anchor. No fake social proof or scarcity — on-brand for an honest-audit product. Renders via next/og (verified 200 PNG).

v0.45.2

Added

  • App version in the footer — the running version (from package.json, exposed via NEXT_PUBLIC_APP_VERSION in next.config.ts) now shows as · v<x.y.z> in the global footer, so it's obvious which build is live.

v0.45.1

Changed

  • Completed the FAQ rollout — added ToolFaq (visible accordion + FAQPage structured data) to the remaining 24 tool pages, so every tool page now has an FAQ (only the /tools index doesn't, as it's a listing). Better AEO/rich-result coverage and thicker content on the lighter utility/generator pages. 305 tests green; build compiles; live-verified schema + visible FAQ render (incl. escaped-quote content).

v0.45.0

Changed / Fixed (SEO & GEO hardening — from the critical audit)

  • Restored static rendering. Reverted the per-request nonce CSP (0.43.0) and the force-dynamic root layout back to a static CSP baseline, so tool pages and guides are prerendered/CDN-cacheable again — build went from 44 dynamic / 2 static to ~29 static / 20 dynamic (the remainder are API routes and genuinely dynamic pages). Recovers TTFB, LCP and crawl efficiency; the static CSP keeps frame-ancestors 'none', object-src 'none' etc. (the app has no script-injection sink, so 'unsafe-inline' is an acceptable trade for static generation).
  • Site-level structured data. Added global Organization + WebSite JSON-LD (with SearchAction pointing at the domain audit) in the root layout — the brand entity search and AI systems need. Previously the site had none.
  • Default social card. Added a branded 1200×630 opengraph-image (auto-applied to every page) plus a default twitter:card = summary_large_image and og:site_name. The site no longer ships imageless share links.
  • Icons & manifest. Added dynamic icon, apple-icon and manifest.webmanifest; removed the leftover create-next-app files (next.svg, vercel.svg, file/globe/window.svg) from public/.
  • Titles. Trimmed 15 page titles that exceeded ~60 characters so the keyword-rich part isn't truncated in search results.
  • Homepage metadata. Added a canonical, a keyword-rich description and Open Graph for / (it previously only inherited the layout defaults).
  • FAQ structured data. New self-contained ToolFaq component (visible accordion + FAQPage JSON-LD); added FAQs to 16 key tool pages (audit, AI Visibility/Readiness, deliverability, CWV, SSL, DNS, HTTP headers, security, SPF, DMARC, WHOIS, subdomains, link preview, robots, sitemap). toolJsonLd already emitted BreadcrumbList for every tool. — 305 tests green; production build compiles; verified live: homepage emits Organization+WebSite+SearchAction, OG image serves a 200 PNG, tool pages carry FAQPage schema + visible FAQ.
  • Not code-fixable (operational): the only deployment is behind Vercel SSO and crawlsonar.com isn't connected yet, so nothing is indexable until a public production deploy; wire Google Search Console + analytics after launch.

v0.44.0

Added

  • Guides rebuilt into a comprehensive knowledge base — the go-to reference behind the tools, engineered to be maximally search- and AI-friendly. Replaced the 4 hand-written guide pages with a data-driven engine: each guide is a structured object (src/content/guides/*.ts) rendered by one template (src/components/guide-article.tsx) that auto-generates, for every guide: — Structured data: TechArticle + BreadcrumbList + FAQPage + optional HowTo JSON-LD (verified: the GEO pillar emits all four). — An anchored table of contents, reading time, category, published/updated dates, semantic headings with deep-link anchors, callouts, tables, code and step lists, an FAQ accordion, and dense internal linking to related tools and guides.
  • 15 in-depth guides across 6 topics (up from 4): AI readiness & GEO (Generative Engine Optimization, llms.txt, SSR-for-AI, controlling AI crawlers), Email (deliverability pillar, SPF, DMARC, DKIM), SEO (structured data, Open Graph), Security (security headers, CSP, SSL errors), DNS (records explained, DNSSEC), Performance (Core Web Vitals). Each ends with the free tool to check your own site.
  • A categorized /guides hub with CollectionPage JSON-LD, and the full set surfaced in sitemap.xml (with real updated dates) and llms.txt (categorized) for AI discovery. — Build compiles (52 routes); 305 tests green; /guides/<slug> renders with TOC + FAQ + 4 schemas, /guides lists all 15, unknown slugs 404. Old static guide routes removed (migrated into data).

v0.43.0

Security

  • Strict, nonce-based CSP — completed the hardening flagged in 0.42.0. src/proxy.ts now issues a fresh per-request nonce and sets script-src 'self' 'nonce-…' 'strict-dynamic', so 'unsafe-inline' (and 'unsafe-eval') are gone from script-src in production. Next.js reads the nonce from the request CSP header and stamps it on its own inline scripts. application/ld+json is data (not gated by script-src), so structured data stays intact with no nonce. The five static headers (HSTS, nosniff, X-Frame-Options, Referrer-Policy, Permissions-Policy) remain in next.config.ts. — Verified in a production build (next start): CSP nonce and the nonce on Next's scripts match within a single request, no 'unsafe-inline'/'unsafe-eval', JSON-LD present, client hydration + fetches work. 305 tests green. — Trade-off (deliberate): a per-request nonce can't live in a prerendered page, so the root layout is now export const dynamic = "force-dynamic" — the whole app renders at request time (no static/CDN HTML caching). For a free-tools SEO site that's a real speed/cost cost for a defense-in-depth gain (the app has no script-injection sink — React escapes all input; the only dangerouslySetInnerHTML is our own JSON-LD). To restore static generation, revert this commit (drops back to the 0.42.0 'unsafe-inline' baseline).

v0.42.0

Security

  • The app now sends its own security headers on every response (previously it had the header *generator/checker* for users but shipped none itself — it would have failed the audit it runs on others). Added via next.config.ts headers(): Content-Security-Policy, Strict-Transport-Security (2y, includeSubDomains), X-Content-Type-Options: nosniff, X-Frame-Options: DENY, Referrer-Policy: strict-origin-when-cross-origin, a locked-down Permissions-Policy, and X-DNS-Prefetch-Control. — CSP is tuned to the app: frame-ancestors 'none', object-src 'none', base-uri 'self', form-action 'self', img-src 'self' data: https: (the Link Preview tool renders remote og:images), connect-src 'self', upgrade-insecure-requests. script-src/style-src keep 'unsafe-inline' because every tool page emits inline JSON-LD (structured data is core to the product) and Next hydrates inline; dev also allows 'unsafe-eval' + ws: for HMR. — Verified live: all six headers present, no CSP console violations, client fetches (connect-src 'self'), hydration and inline JSON-LD all intact. — Follow-up (tracked): tighten script-src to a per-request nonce (middleware-injected) to drop 'unsafe-inline' — the last hardening step.

v0.41.0

Added

  • AI Crawler Log Analyzer (/tools/ai-crawler-log) — the observational counterpart to the robots.txt generator: upload or paste a web-server access log to see which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, CCBot, Bytespider, Google-Extended and ~15 more) actually visited, how often, and whether they were blocked (403/401) — which quietly makes you invisible to those AI systems. [Tier 2] — Fully client-side (same privacy stance as the Email Header Analyzer): the log is parsed in the browser and never uploaded. Pure analyzeCrawlerLog / parseLogLine (combined-log UA extraction, bot matching, blocked-hit detection, sorting) are unit-tested. 305 total tests. Build 86/86. Live-verified: 4 AI hits across GPTBot (2, incl. one 403), ClaudeBot and PerplexityBot, human traffic ignored.

v0.40.0

Added

  • Compare Two Websites (/tools/compare) — head-to-head audit: runs the full Domain Audit on two domains in parallel and scores them module by module, highlighting the winner of each check and overall. Turns an abstract score into a competitive to-do list; naturally shareable. Reuses runDomainAudit; pure compareAudits(a, b) diff is unit-tested (per-module winner, ties, ran-only-one-side, overall). [Tier 3] — Route maxDuration = 60, tight rate limit (two audits). 300 total tests. Build 85/85. Live-verified: github.com 74 vs gitlab.com 73 in 3.7s, with a sensible per-module breakdown (ties handled).

v0.39.0

Added

  • Public developer API — every check from one authenticated endpoint (POST /api/v1/run with { tool, url }), plus a docs page at /api-docs with a one-click free-key generator, curl example, the tool table and limits. Opens integrations, CI and agent workflows (and the paid-tier upsell). [Tier 3] — Accountless API keys: POST /api/v1/keys mints a cs_live_… key, shown once; only its SHA-256 hash is stored (KV in prod, filesystem in dev). Auth via Authorization: Bearer or x-api-key. Per-key daily quota metered in KV (free tier 100/day, API_FREE_DAILY_LIMIT) with x-ratelimit-* headers, plus a per-IP burst cap and a mint cap. — A single tool registry (src/lib/api/registry.ts) maps 13 tool ids to the existing check modules (domain-audit, dns, ssl, http-headers, whois, email-deliverability, ai-readiness, llms-txt, performance, core-web-vitals, security, subdomains, link-preview), normalising input per tool — so the API and the UI share one implementation. Typed check errors are mapped to sensible HTTP statuses; every call stays SSRF-safe. — 8 new unit tests (key mint/hash, header extraction, auth/metering round-trip, registry). 297 total. Build 83/83. Live-verified: 401 unauthenticated → mint key → authenticated dns run 200 with rate-limit headers → unknown-tool 400 listing the 13 tools.

v0.38.0

Added

  • Website & Domain Monitoring (/tools/monitoring) — the Tier-3 recurring-value engine that turns one-off tools into a reason to come back. Create a monitor for a domain and get alerted on uptime, SSL-certificate expiry (14-day warning), domain-registration expiry (30-day warning) or DNS-record change, via email or webhook. Accountless: creating a monitor returns a private manage link (/monitor/{token}), the same capability-URL pattern as shared reports — no auth to build yet. — Cron (/api/v1/cron/monitor, hourly via vercel.json, CRON_SECRET-gated with a timing-safe check) iterates all monitors, re-runs each check, and dispatches alerts. Alerting is a pure decideAlert(type, prevState, prevAlarm, outcome) function (first-run seeding, threshold enter/leave, change detection) — fully unit-tested — so it only fires on a genuine change and again on recovery (no spam). — Checks reuse the existing TLS / WHOIS / DNS / HTTP modules. Store is the same KV-in-prod / filesystem-in-dev dual pattern as reports. Webhooks are SSRF-safe: safeFetch gained POST-body support so user-supplied webhook URLs are delivered through the same public-IP-pinned fetcher (can't be pointed at internal services). Email reuses the dormant delivery pattern (logs intent until a provider is wired). — 9 new unit tests (alert-decision matrix). 289 total. Build 80/80. Live-verified full lifecycle: create → seeded check (example.com SSL "51 days") → view → cron re-check (no false alert) → manage page (noindex) → delete → 404.

Note

  • The hourly cron needs a Vercel Pro plan (Hobby caps cron at daily); expiry checks are fine daily, uptime benefits from the hourly cadence. Set CRON_SECRET in production.

v0.37.0

Added

  • AI Visibility Checker (GEO) (/tools/ai-visibility) — the Tier-2 hero and the product's differentiator. Asks an LLM what it knows about a brand from training, compares that to the brand's own site (fetched for the accuracy check), and returns a 0–100 visibility score + grade, whether it's recognised / accurate / positive, and concrete GEO recommendations to get cited in AI answers. — Margin-protected (global "unit economics for AI features" rule): dormant until ANTHROPIC_API_KEY is set (503 otherwise); cheap Haiku model with small max_tokens; 24h per-domain result cache in KV so repeat scans don't re-bill; a global daily budget counter in KV (AI_VISIBILITY_DAILY_BUDGET, default 500) on top of a hard 3-per-hour per-IP limit. — Honest framing throughout: reflects one model's training knowledge (labelled), not a live crawl of every engine; being unknown is common for smaller brands and is what GEO fixes. LLM output is defensively normalised (enum whitelisting, confidence clamp, recommendation cap) so malformed JSON can't break the report. — Added kvIncr / kvExpire to the KV client for the budget counter. Pure transform + normaliser unit-tested (recognised/unrecognised scoring, finding generation, coercion). 280 total tests. Live-verified dormant path (503 without a key) and page render; the LLM happy-path runs once the key is set in prod.

v0.36.0

Added

  • Link Preview & Open Graph Checker (/tools/link-preview) — shows exactly how a URL renders as a card on Google, X/Twitter, Facebook/LinkedIn and Slack/Discord, with the Open Graph / Twitter Card fixes to improve it (missing og:image, over-length title/description, missing twitter:card). Reuses the existing HTML parser; extracts OG + Twitter tags, resolves relative image/canonical/favicon URLs to absolute, and renders four faithful preview cards. Viral, shareable, SEO-adjacent. [Tier 1] — SSRF-safe (page fetched via safeFetch). Pure extractPreview(html, url) transform, unit-tested (OG-over-base fallback, relative-URL resolution, warning generation, favicon default). 274 total tests. — Live-verified against github.com (all four cards render, og:image loads, sensible length warnings). Registered in /tools, sitemap, llms.txt.

Note

  • DNS propagation tool deferred: a meaningful multi-geo propagation checker needs geographically-distributed resolvers, but DoH-JSON over the SSRF port allowlist (80/443/8080/8443) is effectively only Cloudflare + Google (both global anycast → always agree). Not worth shipping a weak version; revisit with a proper resolver source.

v0.35.0

Added

  • Subdomain Finder (/tools/subdomain-finder) — passively enumerate a domain's subdomains and certificate issuers from public Certificate Transparency logs (crt.sh), and flag exposed dev/staging/admin/internal hosts. A bespoke discovery UI (stat tiles, issuer chips, highlighted sensitive hostnames) rather than the scored report shape, since this is enumeration not pass/fail. [Tier 1] — SSRF-safe (only outbound call is to crt.sh via safeFetch); passive — never scans or brute-forces the target. Route maxDuration = 30 (crt.sh is slow). Pure entries→report transform, unit-tested (SAN parsing, wildcard stripping, domain scoping, issuer aggregation, sensitive-host flagging). 270 total tests. — Live-verified against anthropic.com: 126 subdomains from 3,798 certs, correctly surfacing staging, test, internal.api and portal hosts. Registered in /tools, sitemap, llms.txt.

v0.34.0

Added

  • Email Deliverability Test (/tools/email-deliverability) — the sending-authentication trio (SPF, DKIM, DMARC) plus MTA-STS / TLS-RPT / BIMI in one focused, scored report, with the exact fix per failing check. Adds DKIM selector probing (queries ~20 common provider selectors) on top of the SPF/DMARC parsers reused from the DNS module, and weights everything for inbox deliverability (Gmail/Yahoo now require SPF+DKIM+DMARC for bulk senders). [Tier 1] — DNS-only (no SSRF surface). DKIM is selector-based and can't be fully enumerated — the tool probes common selectors and says so, so a custom-selector setup reads as "no DKIM found" with a clear caveat (verified: google.com's rotating date selectors aren't guessable; github.com's google selector is found). — Pure resolved-facts→report transform, unit-tested (healthy vs broken vs no-SPF). 266 total tests. Registered in /tools, sitemap, llms.txt. Live-verified against google.com (82/B) and github.com (91/A).

v0.33.0

Added

  • WHOIS / RDAP Lookup (/tools/whois-lookup) — registrar, registration/expiry dates, registry status, nameservers and DNSSEC for any domain, via the modern RDAP protocol (structured JSON successor to WHOIS). The actionable win is expiry: flags a domain expiring within 30 days (critical if already lapsed), plus any blocking status (hold / pending delete). Feeds the upcoming expiry monitor. [Tier 1] — SSRF-safe: resolved through the rdap.org redirector using safeFetch, which re-validates every redirect hop's IP, so the TLD-dependent final host can't be abused. Falls back to the registrable domain when a subdomain is entered. — 7 new unit tests over a pure RDAP→report transform + input normaliser — 262 total. Registered in /tools, sitemap, llms.txt. Live-verified against google.com and bbc.co.uk (multi-part .co.uk TLD handled).

v0.32.0

Added

  • Core Web Vitals Checker (/tools/core-web-vitals) — the biggest missing SEO/perf signal. Powered by Google PageSpeed Insights: real-user field data (CrUX) for LCP / INP / CLS when available, falling back to the Lighthouse lab result for low-traffic sites, plus the overall Lighthouse performance score. Explainable scoring against Google's official "good" thresholds (LCP ≤ 2.5s, INP ≤ 200ms, CLS ≤ 0.1), one issue+fix per failing metric. This is the first item of the Tier-1 one-stop-shop expansion. — No SSRF surface (Google fetches the target; our only outbound call is to googleapis.com). Works keyless at low volume; GOOGLE_PSI_API_KEY optional for production reliability. Kept out of the synchronous Domain Audit on purpose — Lighthouse is slow (route maxDuration = 60, tight 6-per-5-min rate limit). — 5 new unit tests over a pure PSI→report transform (field-preferred, lab fallback, INP-not-applicable, severity capping) — 255 total. Registered in /tools, sitemap and llms.txt. Live-verified: graceful 429 handling keyless; page renders.

v0.31.0

Added

  • Durable serverless store — Vercel-ready. The three filesystem stores (shareable reports, leads, analytics events) now transparently use Upstash Redis in production, since serverless filesystems are ephemeral/read-only. New zero-dependency Upstash REST client (src/lib/kv.ts, explicit JSON (de)serialisation, 5s timeout) behind the existing store interfaces. Report TTL is now native (SET … EX), so shares self-expire with no sweep/cron. The event list is capped (LTRIM) to bound growth. — Dormant-until-configured (same pattern as origin cloaking / email / admin): with no KV_REST_API_URL+KV_REST_API_TOKEN (or UPSTASH_REDIS_REST_URL/TOKEN), the app keeps using the filesystem stores — so local dev needs no Redis and nothing else changed for it. — 11 new unit tests (mocked-fetch REST wiring, env gating, JSON round-trip) — 250 total. Report share round-trip re-verified live over HTTP on the filesystem path.

Changed

  • next.config.ts: outputFileTracingIncludes force-bundles pdfkit's .afm font-metric files into the PDF route, avoiding the classic ENOENT … Helvetica.afm failure on Vercel.
  • .env.example documents the KV vars and that they're required on Vercel.

Notes

  • The in-memory rate limiter is per-instance on serverless (state isn't shared across Vercel instances) — acceptable at launch; move to Redis if abuse warrants.
  • The Domain Audit route already sets maxDuration = 60 (needs a Vercel Pro plan; Hobby caps functions at 60s but with fewer resources).

v0.30.1

Changed

  • Privacy policy: filled the operator's statutory details from Companies House. Northstar Infinity Works Ltd — company number 17326480 (incorporated 7 July 2026), registered office C/O Elsg Ltd, Regus, Building 2, Marlins Meadow, Watford, England, WD18 8YA. Removed the [company number] / [registered office address] placeholders and the publish-blocker TODO, so /privacy is now legally complete for launch.

v0.30.0

Changed

  • Rebrand to Crawlsonar (crawlsonar.com). The product is now Crawlsonar everywhere it matters: metadata title/template, llms.txt, PDF report brand, shared-report Open Graph images, User-Agent strings (Crawlsonar/0.1 (+https://crawlsonar.com; …)), JSON-LD publisher/author, the AI robots.txt generator's output comment, and the privacy policy. Default site URL is now https://crawlsonar.com (still overridable via NEXT_PUBLIC_SITE_URL).
  • Global site chrome — added a sticky header (sonar wordmark + Tools / Guides / Domain-audit nav) and a shared footer across every page, so all ~40 tool pages now carry consistent branding and a way back home. Removed the homepage's bespoke footer (the global one replaces it).
  • Privacy policy contact set to [email protected]; .env.example example domains updated to crawlsonar.com (site URL, abuse, from-address).

v0.13.0

Added

  • Share + PDF + email-capture for the Domain Audit — the most shareable artefact is now shareable. The /audit/domain report gets the "🔗 Share this report" button (tokenised /r/{token} link, authentic-by-construction re-run), the PDF lead magnet and email capture, exactly like the other tools. — Extended the report store to a three-way discriminated union (ai-readiness | llms-txt | domain-audit) via a new ReportPayload type; POST /api/v1/reports and POST /api/v1/leads re-run runDomainAudit server-side for audit shares. — PDF renderer now handles the audit shape (overall score/grade header, per-module score breakdown, merged issue list from every module). — Extracted an AuditBody presentational component shared by the live audit page and the read-only /r/{token} page, so the two can't drift.
  • 1 new unit test (audit PDF renders a valid %PDF) — 189 total. Verified live end-to-end: shared a github.com audit → /r/{token} renders (200, noindex, audit body) → 5.4 KB PDF with full module breakdown + all 9 issues.

v0.12.0

Added

  • Full Domain Audit (/audit/domain + POST /api/v1/audit/domain) — the product's headline promise: one input, one coherent report (Requirements v1.0 §1/§7/§8). Classifies the input (domain/URL/email → origin+url) and runs all seven modules in parallel via Promise.allSettled with per-module error isolation (a module that can't run — e.g. TLS on an HTTP-only host — is shown but excluded from the score, so one failure never sinks the report). Aggregates an impact-weighted overall score (AI 25, HTTP 18, TLS 15, DNS 15, robots/sitemap/llms 9 each), severity counts and a merged, severity-sorted issue list; each module card links to its dedicated tool prefilled with the target. Stricter rate limit (5 / 5 min) and maxDuration: 60.
  • Homepage secondary CTA to the audit, sitemap + own llms.txt entries. The audit reuses every existing module — no new checks, pure composition.
  • 8 new unit tests (input classification, grade mapping, weighted aggregation excluding errored modules, severity tallies) — 188 total. Verified live: github.com 77/B (~2.6s), stripe.com 83/B (~3.4s), all 7 modules running; invalid input → 400.

v0.11.0

Added

  • robots.txt Checker (/tools/robots-txt-checker + POST /api/v1/tools/robots): reuses the shared robots parser + AI-crawler matrix to detect full-site blocks (critical), missing Sitemap directives, and whether an explicit AI crawler policy exists; shows the parsed AI matrix and raw file. Clarifies robots.txt controls crawling, not indexing.
  • Sitemap Checker (/tools/sitemap-checker + POST /api/v1/tools/sitemap): reuses parseSitemapXml to confirm sitemap.xml exists and is valid, counts URLs (following the first child of a sitemap index) and samples up to 6 URLs (SSRF-safe fetch) for reachability.
  • Both wired: homepage (all "coming soon" cards now live — 11 tools total), sitemap, own llms.txt, cross-links.
  • 4 new unit tests (robots analysis: absent, healthy, full-site block, missing sitemap/AI policy) — 180 total. Verified live: github.com robots 66/C (no sitemap directive, no AI policy), vercel.com sitemap 100/A (500 URLs, 0 broken), sites without a sitemap → no_sitemap. Confirmed SSRF still blocks localhost self-scan.

v0.10.0

Added

  • SSL / TLS Checker (/tools/ssl-checker + POST /api/v1/tools/ssl). Inspects the certificate: validity window, days-to-expiry (with expiring-soon warnings), hostname match (SAN/CN incl. single-label wildcards), chain trust vs default roots, negotiated protocol (flags < TLS 1.2), key strength, plus the full chain and connected IP. SSRF-safe: resolves the host, validates every IP with isPublicIp, and connects to the pinned IP with servername set for SNI/identity. Pure cert-evaluation helpers (SAN parse, wildcard match, validity) are unit-tested.
  • 11 new unit tests — 176 total. Verified live across badssl.com fixtures: cloudflare.com 100/A, expired.badssl.com → cert_expired, wrong.host → cert_hostname_mismatch, self-signed → cert_untrusted.

Fixed

  • Key-strength false positive: ECDSA certificates (e.g. Cloudflare's P-256) were wrongly flagged as weak keys because the check assumed RSA. Now ECDSA keys are recognised as strong; only RSA < 2048 is flagged. Caught during live verification (cloudflare.com went from a false 90 to a correct 100/A).

v0.9.0

Added

  • HTTP Header Checker (/tools/http-headers-checker + POST /api/v1/tools/http-headers). Through the SSRF-safe fetcher: redirect chain with per-hop status, HTTPS-final + HTTP→HTTPS-upgrade probe, security-header inspection (HSTS, CSP, X-Frame-Options/frame-ancestors, X-Content-Type-Options, Referrer-Policy, Permissions-Policy, COOP/COEP/CORP), cookie-flag analysis (Secure/HttpOnly/SameSite) and server/framework version-leak detection. Header analysis is a pure, unit-tested function. Redirect-chain view + security-header table + issues. Homepage card, sitemap, own llms.txt, cross-links.
  • 7 new unit tests (Set-Cookie parsing, secure vs insecure response analysis, cookie flags, long redirect chain) — 165 total. Verified live: github.com 89/B (HSTS+CSP+X-Frame+nosniff+Referrer, flags one insecure cookie); insecure HTTP target correctly scored, and slow targets return a clean 502 on timeout (no crash).

v0.8.0

Added

  • DNS Lookup (/tools/dns-lookup + POST /api/v1/tools/dns) — first sprint-3 commodity tool (internal-link hub). Resolves A/AAAA/CNAME/MX/TXT/NS/SOA/CAA via node:dns with bounded timeouts, parses SPF (with a 10-DNS-lookup-limit count) and DMARC (policy strength). Issue detection: multiple SPF records (critical), SPF over 10 lookups (high), missing SPF/DMARC (medium), p=none (low), missing CAA (low), no address records (high). Records tables, email-authentication panel, explainable score. Input accepts a bare domain, URL or email address (domain extracted). Homepage card (DNS no longer "coming soon"), sitemap, own llms.txt, cross-links.
  • 9 new unit tests (SPF lookup counting, SPF/DMARC parsing incl. case + multiple records) — 158 total. Verified live: github.com 94/A, email-address extraction ([email protected] → stripe.com), invalid input → 400.

v0.7.0

Added

  • AI robots.txt Generator (/tools/ai-robots-txt-generator) — the actionable other half of the AI Readiness checker's crawler matrix (Requirements v1.0 Appendix B). Per-crawler Allow/Block toggles grouped by purpose (training / AI search / user-triggered), three presets ("Block training, allow AI search" default, "Allow all", "Block all"), live copyable/downloadable robots.txt output. Honest framing baked in: robots.txt is advisory, and Google-Extended/Applebot-Extended are opt-out tokens (noted only when blocked). Pure client-side — no network, no SSRF surface.
  • Cross-linked: the AI Readiness no_ai_crawler_policy issue fix now points at this generator; homepage card, sitemap entry, own llms.txt entry, related-tools links.
  • 7 new unit tests including a round-trip through the checker's own robots.txt parser proving the generator and checker agree — 149 total. Verified live: presets, per-bot toggles, custom state and opt-out-token note.

v0.6.0

Added

  • Email-capture + PDF lead magnet (Roadmap v1.1 sprint 2 — the free→paid bridge). "Get this report as a PDF" appears on both live checkers and on shared /r/{token} pages. — Real PDF, no headless browser: src/lib/pdf/report-pdf.ts renders either report type with pdfkit (header, score, category breakdown, severity-coloured issues, full checks list, footer disclaimer). serverExternalPackages: ["pdfkit"] keeps its font metrics loadable at runtime. Downloadable at GET /api/v1/reports/{token}/pdf (token-validated, rate-limited). Verified live: valid 4.3 KB %PDF with correct content-type/disposition. — GDPR-first capture: explicit consent checkbox is mandatory (consent: z.literal(true); submit disabled until email valid + consent checked). Leads stored minimally (email, tool, target, consent timestamp, no IP) in a swappable filesystem LeadStore (.data/leads/leads.jsonl). POST /api/v1/leads re-runs the scan server-side to persist an authentic report, stores the lead, and returns a token + PDF URL so the download works immediately. — Dormant email delivery (src/lib/leads/email.ts): active only when EMAIL_API_KEY + EMAIL_FROM are set (provider send is a launch TODO); until then it logs intent (domain only, never the full address) and the UI honestly offers the download without claiming an inbox delivery.
  • 7 new unit tests (PDF buffer validity, email validation, lead store no-IP, dormant-email gating) — 142 total. Verified live end-to-end: consent enforced (400 without), invalid email rejected (400), lead stored without IP, dormant email logged, PDF rendered.

v0.5.0

Added

  • Shareable reports (Roadmap v1.1 sprint 2 + third-party report policy). "🔗 Share this report" on both checkers creates a tokenised link at /r/{token}. — Authentic by construction: POST /api/v1/reports RE-RUNS the scan server-side rather than accepting client JSON, so a shared report about someone else's site cannot be fabricated. — Security: 128-bit crypto-random base64url tokens; every filesystem path gated by a strict token regex (path-traversal-proof, verified live with ..%2F..%2Fetc%2Fpasswd → 404); 30-day expiry with lazy-delete-on-read; 512 KB size cap; report-creation rate limit (12 / 5 min per IP). — Policy: /r/* pages are noindex, nofollow and disallowed in robots.txt; visible point-in-time disclaimer ("not a security certification"); owner opt-out contact via optional NEXT_PUBLIC_ABUSE_EMAIL; reports store only public target data (no visitor IP/headers).
  • Report store (src/lib/reports/) as a swappable ReportStore interface with a filesystem implementation (.data/reports, gitignored). NOTE documented: swap for object store / DB before any serverless deploy (ephemeral FS).
  • Shared presentational components (src/components/report/): ScoreCard, IssueList, ChecksList, CategoryBars, CrawlerTable, AiReadinessBody, LlmsTxtBody. Both live checkers refactored to use them, so the live and shared-report views can never drift.
  • 7 new unit tests (token validation incl. traversal vectors, store round-trip, expiry deletion) — 135 total. Verified live end-to-end: create link in UI → open /r/{token} → renders, noindex, disclaimer, report_viewed{shared:true} event.

v0.4.0

Added

  • AI Readiness Checker (/tools/ai-readiness-checker + POST /api/v1/tools/ai-readiness) — the flagship differentiator (Roadmap v1.1 sprint 2). Scores a page across seven weighted categories summing to 100 (Requirements v1.0 §17): crawlability (20), content extraction (20), entity clarity (15), structured data (15), trust signals (10), AI crawler governance (10), agent readiness (10). Per-category breakdown bars, evidence+fix issue cards, an AI crawler policy matrix (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Google-Extended, PerplexityBot, CCBot, Bytespider, Applebot-Extended → allowed/blocked/partial/unspecified from robots.txt), and a markdown extraction preview ("what an LLM sees"). Analysis is on server-rendered HTML only (no JS) — stated honestly in the UI, since that's what crawlers/LLM fetchers see.
  • Supporting modules: robots-rules.ts (robots.txt parser with Google-style longest-match path matching, per-agent group selection, wildcard/$ patterns, AI crawler matrix) and html-analysis.ts (main-content extraction + noise ratio + markdown, heading outline, JSON-LD parsing, form-label a11y, link-text quality, alt coverage, page signals).
  • Homepage hero now routes to the AI Readiness Checker; new tool card, sitemap entry, own llms.txt entry, and cross-links between all four tools.
  • 24 new unit tests (robots matching/matrix, HTML extraction/JSON-LD/a11y). Caught two real bugs before ship: JSON-LD blocks were unreadable because parseHtml dropped <script> text, and a noise-ratio fixture was miscalibrated. 128 tests total. Verified live against MDN (no JSON-LD → schema 0) and stripe.com (WebSite+Organization → schema 15/15).

v0.3.0

Added

  • llms.txt Generator (/tools/llms-txt-generator + POST /api/v1/tools/llms-txt-generator): builds a draft llms.txt from sitemap.xml (sitemap-index support, 500-URL cap) or homepage nav links as fallback, with H1/summary from homepage title/og/meta description. URL categorisation into Documentation / API / Products / Pricing / Blog / Company / Legal sections; per-section (12) and total (40) link caps keep output curated. Editable textarea, copy + download, "this is a draft — curate it" guidance. Full SEO page template; cross-linked with the llms.txt Checker both ways (checker's missing-llms.txt fix now points at the generator). Verified live: vercel.com (501 sitemap URLs → 40 links, 7 sections) and llmstxt.org via UI.
  • 20 new unit tests (sitemap XML parsing, URL categorisation, slug humanisation, section capping, output structure) — 104 total.

v0.2.0

Added

  • Origin cloaking behind Cloudflare (dormant until launch): src/proxy.ts gate + src/lib/security/origin-auth.ts. When ORIGIN_SHARED_SECRET is set, every request must carry the Cloudflare-injected x-origin-auth header or the app answers a bare 403 — direct hits on the origin get nothing. Constant-time comparison, minimum 32-char secrets, comma-separated dual secrets for zero-downtime rotation. Unset in dev = no-op. Verified live: dormant 200; active → no header 403, wrong header 403, secret #1 200, rotated secret #2 200, API without header 403.
  • Rate limiter now keys on cf-connecting-ip when (and only when) origin cloaking is active — behind Cloudflare the leftmost x-forwarded-for value is client-controlled and would allow rate-limit evasion.
  • .env.example: ORIGIN_SHARED_SECRET documentation with generation command and launch-time reminder.
  • 11 new unit tests (84 total).

v0.1.0

Added

  • Next.js 16 + TypeScript app (App Router, Tailwind 4, React 19) — modular monolith per Roadmap v1.1 principle 5.
  • SSRF-pinned safe fetcher (src/lib/security/): IP validation blocklist (RFC1918, link-local/cloud-metadata 169.254.0.0/16, CGNAT, multicast/reserved, IPv6 loopback/link-local/ULA/site-local, IPv4-mapped/NAT64/6to4/Teredo embedded forms), validation inside the DNS lookup callback so the validated address IS the connected address (atomic pin — no TOCTOU/rebinding window), whole-fetch refusal when any resolved record is non-public, manual redirect handling with per-hop revalidation, scheme+port allowlists, body-size and time caps. Verified live: metadata IP, decimal-encoded 127.0.0.1 and 127.0.0.1.nip.io rebinding all blocked.
  • Common issue schema + explainable scoring (src/lib/issues/): severity/category/impact/evidence/fix per Requirements v1.0 §22; critical/high issues cap module scores; not-applicable checks excluded from scoring.
  • llms.txt checker (src/lib/checks/llms-txt.ts + /tools/llms-txt-checker): availability, content-type/HTML-fallback detection, llmstxt.org structure validation (H1, summary blockquote, sections, links), link-liveness sampling, thin-file detection, llms-full.txt check. Full SEO page template: tool above fold, explanation with honest "Google ignores llms.txt" note, example, common problems, FAQ, unique metadata, canonical, OG, JSON-LD (WebApplication + BreadcrumbList).
  • API POST /api/v1/tools/llms-txt: zod validation, per-IP rate limiting (10/min anonymous, verified live), no internal error leakage.
  • First-party cookieless funnel analytics: scan_started/scan_completed/report_viewed (+ reserved names for the full funnel) via sendBeacon to /api/v1/events; no cookies, no IP, no PII stored.
  • Homepage with one-input-first hero routing into the checker; product's own llms.txt, robots.txt, sitemap.xml (dogfooding, launch DoD).
  • Tests: 73 unit tests (SSRF IP validation incl. negative vectors, llms.txt parser fixtures, scoring) — all passing; production build clean.

v0.29.0

Added

  • Privacy & Data Protection Policy (/privacy) — a UK GDPR / Data Protection Act 2018 policy with Northstar Infinity Works Ltd as the named data controller, tailored to what the Service actually processes: cookieless analytics with no IP, client-side-only email-header parsing, consent-based lead capture, transient IP for rate-limiting, 30-day shared reports, processors (hosting / Cloudflare / email provider), international transfers, retention, data-subject rights and the ICO complaint route. Linked from the email-capture consent notice, a new homepage footer, and the sitemap. (Draft — placeholders for company number, registered office and contact email to be filled; legal review recommended before launch.)

v0.28.1

Fixed

  • Content-negotiation bug: HTML analysers were served markdown. The safe fetcher's default Accept header preferred text/plain, text/markdown over text/html. On sites that content-negotiate (e.g. AI-optimised builds that serve a clean markdown alternate to markdown-preferring clients — ukimmigration.law does exactly this), our HTML analysers received markdown with no <head>, and wrongly reported missing meta description, missing title and no structured data — penalising sites that are, ironically, *more* AI-ready. Changed the default Accept to prefer HTML (text/html,application/xhtml+xml,…). Affected the AI Readiness, On-Page SEO, Structured Data, Security, Privacy checkers and the llms.txt Generator's homepage-metadata fetch. Found via a real-world scan reported by the user.
  • Verified: ukimmigration.law went from a false low score to 89/B with JSON-LD (LegalService + WebSite) correctly detected; the llms.txt checker (which fetches static .txt) is unaffected. 239 tests still pass.

v0.28.0

Added

  • Admin dashboard (/admin, Requirements v1.0 §28 — "admin scan monitoring"). Token-gated, dormant-until-configured: with no ADMIN_TOKEN the admin area does not exist (routes 404); with a token set it requires sign-in. Security-by-design: httpOnly + secure + sameSite=strict session cookie, constant-time token comparison (reusing the origin-cloaking primitive), rate-limited login (5 / 5 min), noindex + robots-disallowed, and it sits behind origin cloaking when that's enabled. Read-only dashboard showing the funnel (event counts), scans by tool, lead count + recent leads, and shared-report count.
  • Analytics events are now persisted to a filesystem store (.data/events/events.jsonl, no IP/PII) in addition to stdout, feeding the dashboard; readLeads added. NOTE documented: swap the filesystem stores for a DB before serverless.
  • 4 new unit tests (admin enabled-state + constant-time verification) — 239 total. Verified live end-to-end: 404 without token; 401 on wrong token; login sets the cookie; dashboard renders the persisted funnel data.

v0.27.0

Added

  • DNSSEC validation + DNSKEY/DS + HTTPS/SVCB records over DNS-over-HTTPS (Requirements v1.0 §10 — completes the record coverage node's resolver can't reach). New src/lib/checks/doh.ts queries Cloudflare DoH (application/dns-json) through the SSRF-safe fetcher; a pure parseDohJson + deriveDnssec are unit-tested. The DNS Lookup now reports DNSSEC status (validated / signed / unsigned, from the Authenticated-Data flag plus DNSKEY/DS presence) and whether an HTTPS/SVCB record exists, both surfaced in the tool and scored (unsigned → a low DNSSEC issue). Verified live: cloudflare.com DNSSEC validated (AD + DNSKEY + DS), HTTPS record present.
  • 7 new unit tests — 235 total.

v0.26.0

Added

  • DNS depth (Requirements v1.0 §10): the DNS Lookup now also detects BIMI, MTA-STS and TLS-RPT records (advanced email hardening), surfaced in the tool's email panel and scored for mail-enabled domains. Verified live: google.com → MTA-STS + TLS-RPT present.
  • SSL depth (§12): the SSL Checker now explicitly probes TLS 1.2 and TLS 1.3 support (two extra version-pinned handshakes) and flags a missing TLS 1.3. Shown in the certificate panel. Verified live: cloudflare.com supports both.

Deferred (documented in project backlog)

  • Deeper DNS records (DNSKEY/DS/TLSA/SVCB/HTTPS) and DNSSEC validation need DNS-over-HTTPS (node's resolver can't fetch those types) — parked. IP ASN/geolocation + VPN/Tor detection need a paid data provider (MaxMind/IPinfo). Field Core Web Vitals need a headless browser. OpenAPI checker + llms-full/OpenAPI-starter/MCP generators are lower-value/phase-3.

v0.25.0

Added

  • Three more generators (Requirements v1.0 Appendix B), pure client-side builders + tests: — Open Graph Generator (/tools/og-tag-generator): og: + twitter: meta tags for rich link previews, XML-escaped. — CAA Generator (/tools/caa-generator): DNS CAA records (issue/issuewild + iodef), including the "disallow all" case. — MTA-STS & TLS-RPT Generator (/tools/mta-sts-generator): the policy file plus both TXT records, with a testing→enforce rollout and a minimum max_age.
  • 6 new unit tests — 228 total. All three pages verified 200; wired into the /tools Generators group, sitemap and llms.txt.

v0.24.0

Added

  • On-Page SEO Checker (/tools/seo-checker, Requirements v1.0 §15): title length, meta description, canonical, indexability, single H1 + heading order, lang, mobile viewport, Open Graph, image alt coverage and anchor-text quality — reusing the shared HTML-analysis helpers, framed for SEO. Verified live: stripe.com 85/B.
  • Privacy & Cookie Scanner (/tools/privacy-scanner, Requirements v1.0 §21): third-party tracker detection, cookie Secure/HttpOnly/SameSite flags, privacy-policy link and forms-over-HTTPS — with risk wording, not legal conclusions, and an honest note that consent-timing needs a real browser. Verified live: theguardian.com correctly reports policy link + insecure cookies.
  • ReportScanClient — a reusable, RSC-safe generic client (serialisable string props only) for standard {score, issues, checks} tools, used by both new pages. Two thin API routes generated from the shared template. 222 tests (both reuse already-tested analysis helpers; orchestration verified live).

v0.23.0

Added

  • Website Speed Test (/tools/performance + POST /api/v1/tools/performance, Requirements v1.0 §20 MVP). SSRF-safe timed request (resolve → validate → connect to the pinned IP) captures DNS / TCP / TLS / TTFB / total via socket timing hooks, plus HTML size, compression (gzip/br/zstd), cache-control and CDN detection (Cloudflare/CloudFront/Fastly/Akamai/Vercel/Netlify). Timing waterfall bars + issues (slow TTFB, no compression, large HTML). Field Core Web Vitals (LCP/CLS/INP) are explicitly out of scope — they need a real browser (Phase 2, noted in the UI). Verified live: cloudflare.com TTFB 141ms (CDN detected), github.com 164ms (gzip).

v0.22.0

Added

  • Structured Data Checker (/tools/structured-data-checker + POST /api/v1/tools/structured-data, Requirements v1.0 §16): fetches a page, extracts and validates all JSON-LD (reusing the AI-readiness parser), lists the schema.org types, flags invalid JSON, and notes Microdata/RDFa presence. Verified live: stripe.com 100/A (WebSite + Organization).
  • JSON-LD Generator (/tools/schema-generator, Appendix B): pure client-side builder for Organization, WebSite (with SearchAction), Article, LocalBusiness (PostalAddress) and Product (Offer) from a form; omits empty fields, outputs a ready <script type="application/ld+json"> block. Cross-linked with the checker and the structured-data guide.
  • 5 new unit tests (JSON-LD building per type incl. conditional Offer/address, valid-JSON render) — 222 total.

v0.21.0

Added

  • Website Security Scanner (/tools/security-scanner + POST /api/v1/tools/security-scan, Requirements v1.0 §19). Passive checks plus a few light probes of well-known paths, all through the SSRF-safe fetcher with a tighter rate limit (8/min): exposed .git (critical) and .env (critical), mixed content on HTTPS, CORS wildcard-with-credentials, server/framework version leakage, security.txt presence, WordPress REST exposure, and third-party tracker detection (GA/GTM/Meta/TikTok/Hotjar/LinkedIn). UI and AUP disclaim the light active probing ("only scan sites you own or are authorised to test"). Pure findMixedContent + detectTrackers helpers unit-tested.
  • 4 new unit tests — 217 total. Verified live: github.com 100/A (clean, has security.txt); a tracker-heavy site correctly flagged Google Analytics + missing security.txt.

v0.20.0

Added

  • Guides / fix-content (Tier 3 — topic clusters + AdSense prep). New /guides index plus four written, cross-linked articles for the most common issues our tools flag: fixing multiple SPF records, adding HTTP security headers (HSTS/CSP), adding structured data (JSON-LD), and server-side rendering for AI. Each guide has Article JSON-LD, unique metadata, and links to the relevant tool/generator; a shared GuideLayout keeps them consistent. Homepage, sitemap (index + articles) and own llms.txt updated. All 5 pages statically generated and verified 200.

v0.19.0

Added

  • Embeddable AI Readiness badge (Tier 3 — backlink growth loop). /tools/badge runs a full audit, then hands you HTML and Markdown snippets for a shields-style score badge that links back to a live audit of your domain — the classic SSL-Labs/GTmetrix backlink + brand loop. The badge is a lightweight SVG served from GET /api/v1/badge?label&score&grade, coloured by score and hard-cached (content fully determined by query params). Inputs are sanitised and clamped (score 0–100, label stripped, XML-escaped) to prevent SVG/markup injection — verified live: <script> label → "script", score 999 → 100.
  • Added to the /tools Utilities group and sitemap. Verified live: valid SVG renders as a green "AI Readiness 83/100 (B)" badge; /tools/badge returns 200.

v0.18.0

Added

  • Generators pack (Requirements v1.0 Appendix B) — three client-side, deterministic generators (pure builder libs, no network): — Security Headers Generator (/tools/security-headers-generator): HSTS (max-age/includeSubDomains/preload), CSP preset (strict/basic), X-Frame-Options, nosniff, Referrer-Policy, Permissions-Policy, output as raw headers or nginx/Apache config. — SPF Generator (/tools/spf-generator): builds a single v=spf1 record from mx/a/ip4/ip6/includes with a live 10-DNS-lookup-limit count. — DMARC Generator (/tools/dmarc-generator): policy, subdomain policy, pct, rua/ruf, adkim/aspf alignment.
  • Each cross-links to its matching checker; new Generators group on /tools; sitemap + own llms.txt updated.
  • 7 new unit tests (header building incl. nginx/apache render, SPF assembly + lookup count, DMARC assembly) — 213 total. Verified live: Security Headers Generator produces correct HSTS/CSP/X-Frame output and switches to nginx add_header … always; format.

v0.17.0

Added

  • Email Header Analyzer (/tools/email-header-analyzer, Requirements v1.0 §11) — paste raw email headers to see SPF/DKIM/DMARC pass/fail (from Authentication-Results), the delivery path with per-hop delays, sending IP and core message fields, with issues and fixes ("why did my email go to spam"). Fully client-side — the pure src/lib/email-headers.ts parser (header unfolding, auth-result extraction, Received-hop ordering + delay calc) runs in the browser, so pasted headers never reach our server (§11's "never expose pasted headers", taken to its logical end). Cross-links to the SPF/DMARC checkers.
  • 7 new unit tests (header unfolding + body cut-off, auth-result parsing incl. softfail, hop ordering/delay/sending-IP, full analysis pass/fail cases) — 206 total. Verified live: failing SPF/DMARC headers score 45/D with correct issues and sending IP.

v0.16.0

Added

  • 8 programmatic SEO landing pages (Requirements v1.0 Appendix A), each targeting a specific high-volume keyword and reusing an existing engine — no new checks, just focused views + tailored content: — SPF Checker, DMARC Checker, MX Lookup, CAA Checker — a shared DnsFocusClient renders only the relevant slice of the DNS report (record + focused issues) and links to the full DNS report. — Redirect Checker — reuses the HTTP-headers engine to show just the redirect chain. — User-Agent Parser — fully client-side (parseUserAgent extracted to a Node-free src/lib/user-agent.ts), decodes browser/OS/device with a bot flag. — What Is My IP — dedicated page around the IP widget.
  • /tools index page grouping every tool (Full audit / AI & SEO / DNS & email / Security & HTTP / Utilities). Homepage now links to it; sitemap rewritten data-driven with all 19 routes; own llms.txt lists the new tools; shared toolJsonLd helper for landing-page structured data.
  • Verified live: all 8 new pages return 200; SPF Checker shows github.com's record with the 8/10-lookup count. 199 tests still green (new pages are presentational over tested engines).

v0.15.0

Added

  • Dynamic Open Graph images for shared reports — every /r/{token} link now unfurls as a branded score card (tool, target, big score, grade, issue count, accent-coloured by score) on X / Slack / LinkedIn / WhatsApp. Multiplies the reach of the existing share mechanic — more unfurl clicks → more scans → more email capture. Rendered via next/og (opengraph-image.tsx, Node runtime so it can read the report store); og:image meta auto-wired plus twitter: summary_large_image. Works despite the page being noindex (unfurls ignore indexing). Verified live: 1200×630 PNG, HTTP 200, correct score/grade/issues.

v0.14.0

Added

  • "Your IP address" widget on the homepage (Requirements v1.0 §9). Shows the visitor's public IP (with IPv4/IPv6), parsed browser + OS, and a copy button. Reads the IP from proxy headers (cf-connecting-ip first — trustworthy behind our Cloudflare proxy — then x-real-ip/x-forwarded-for), so it shows the real public IP in production; on a direct/localhost connection it honestly notes the loopback/private address instead of pretending. GET /api/v1/ip endpoint. Pure pickClientIp + parseUserAgent helpers.
  • 10 new unit tests (IP header priority incl. IPv6 and XFF hop selection; user-agent parsing for Chrome/Edge/Firefox/Safari across macOS/Windows/Linux/iOS) — 199 total. Verified live: browser shows its real Chrome/macOS UA, loopback note on localhost, cf-connecting-ip honoured.

Fixed

  • Dev server: switched npm run dev to next dev --webpack. Next 16 makes Turbopack the default, which on this machine fails to generate the app-router route manifest — every app route returned 404 while static public/ files still served. Webpack dev compiles the routes correctly. (Production next build was always fine.)

Added

  • Revised roadmap: docs/AI_SEO_Network_Toolbox_Roadmap_v1.1.docx (v1.1, 9 July 2026). Supersedes Section 29 of Requirements v1.0. Key changes: inverted build order (AI-readiness wedge ships in sprints 1–2, commodity tools in 3–4, public launch in sprint 2), paid monitoring pulled forward to sprint 5 with AdSense demoted to a revenue floor after it, plus new sections absent from v1.0 — pricing strategy, measurement plan, SSRF fetcher security gates (IP pinning + expanded blocklist), sensitive-data handling, third-party report policy, and a risk register. Architecture directive changed to modular monolith + Postgres queue (pg-boss) instead of 13 worker microservices + Redis.
  • Initialised git repository.
  • Product requirements specification: docs/AI_SEO_Network_Toolbox_Requirements.docx (v1.0, 8 July 2026).
  • CHANGELOG.md (this file) and .gitignore.