Learn
A plain-English knowledge base for everyone building on the web — whether it’s your first site or your hundredth. Every concept behind our checks, explained simply first, then in depth. No jargon, no gatekeeping.
AI & agents
- AI crawlers & robots.txtThe bots that AI companies use to read the web — and the robots.txt file where you decide which ones may read your site.
- llms.txtA simple file at your site root that gives AI tools a curated map of your most important pages.
- Blocking crawlers (Cloudflare / WAF challenges)When your bot protection (like Cloudflare) shows automated visitors a challenge page, it can also block the search and AI crawlers you actually want.
SEO
- Structured data (schema.org / JSON-LD)A machine-readable description of what your page is about, which helps search engines show rich results and helps AI understand your content.
- Canonical URLA tag that tells search engines which address is the ‘official’ one when the same page can be reached by several URLs.
- Meta descriptionThe short summary of a page that search engines often show under its title in results — your chance to earn the click.
Security
- HSTS (HTTP Strict Transport Security)A setting that tells browsers to only ever open your site over the secure HTTPS connection — never plain, interceptable HTTP.
- HTTP security headersSmall instructions your server sends with every page that tell the browser to switch on built-in protections against common attacks.
- SSL / TLS certificates & HTTPSThe technology behind the padlock — it encrypts the connection between your visitor and your site so nobody in between can read or alter it.
- SPF (Sender Policy Framework)A DNS record that lists which mail servers are allowed to send email using your domain, so receivers can spot forgeries.
- DKIM (DomainKeys Identified Mail)A cryptographic signature added to your outgoing email that proves the message really came from your domain and wasn't tampered with.
- DMARCA policy that sits on top of SPF and DKIM, telling receivers what to do with mail that fails authentication — and emailing you reports of who's sending as you.