Access Control — Letting the Right Agents In

The agent-readiness dimension that lets the right agents in and verifies them — Web Bot Auth, robots-for-AI, opt-out tokens and pay-per-crawl signals.

What the access control dimension means

Access control is the agent-readiness dimension that decides which agents may use your site and proves who is actually visiting. Discovery and content make you readable; access control makes you governable. It covers two jobs at once: verifying that a request claiming to be a known agent really is one, and declaring your policy and terms for automated access — from a plain allow/deny to a licensed, paid arrangement. This dimension bridges directly into access economics, where the same signals decide whether you welcome, block or charge AI traffic.

Signals and standards it covers

Web Bot Auth — verifying visiting agents by checking their Ed25519-signed requests under RFC 9421 and publishing your own key directory, so a bot's identity is cryptographic rather than a spoofable user-agent string.
robots.txt for AI — directives that allow or disallow specific AI crawlers and agents by name.
Opt-out and content-signal tokens — machine-readable signals that state how your content may be used (for example for training), so an agent can honor your preference.
Pay-per-crawl and RSL — licensing signals that move access from free/blocked to a metered, paid arrangement for automated fetchers.

Verify the cryptographic facts — RFC 9421, Ed25519, the Signature-Agent header — against the primary IETF and Cloudflare sources at build, never against an internal note.

How the Agent-Readiness Audit scores it

The Audit scores access control on whether your verification and policy signals are present and well-formed. The anchor check is access_control.web_bot_auth: it passes when you publish a valid key directory and your verification path accepts a correctly signed request under RFC 9421. Companion checks confirm a robots policy that names AI agents explicitly rather than relying on defaults, and the presence of any opt-out, content-signal or pay-per-crawl declaration you intend to enforce. Because verification is cryptographic, the pass criterion is unambiguous: a signature either validates against your published key or it does not. This dimension lets you verify the agents catalogued in the crawler registry's verification cluster.

← Agent-Readiness · .md