Implement Web Bot Auth

Verify visiting agents by checking their Ed25519-signed requests under RFC 9421 and publishing your own key directory for others to check.

What Web Bot Auth is

Web Bot Auth is a cryptographic way to prove which bot is which. A legitimate agent signs its HTTP requests with an Ed25519 key and sends a signature using the HTTP Message Signatures standard (RFC 9421), identifying its operator through a Signature-Agent header and a published key directory. Instead of trusting an easily spoofed User-Agent string or maintaining brittle IP allowlists, a site can verify the signature against the operator's keys and know the request is genuinely from that agent.

Why it matters for agents

Access control only works if you can tell a real agent from an impersonator. User-Agent strings are trivially faked and IP ranges drift, so policy built on them is unreliable. Web Bot Auth gives you a verifiable identity to attach policy to — allow this verified agent, rate-limit that one, charge another via pay-per-crawl. It is the verification backbone of the access-control dimension and the same mechanism that powers the Crawler Registry's verification cluster.

How to implement it

Implement Web Bot Auth on two sides — as a verifier of incoming agents, and (if you operate one) as a signer that publishes a key directory.

To verify incoming agents, parse the request's Signature and Signature-Agent headers, fetch the operator's published Ed25519 key directory, and validate the signature per RFC 9421 before applying your access policy.
To be verifiable yourself, generate an Ed25519 keypair, publish the public key in a directory at the spec's declared location, and sign your own outbound agent requests.
Verify: send a signed request and confirm validation passes, and that an unsigned or tampered request is correctly rejected.

This satisfies the audit check access_control.web_bot_auth — pass criterion: a signed request validates against a reachable key directory under RFC 9421. This site publishes its own Web Bot Auth key directory as live proof. Verify the RFC 9421, Ed25519, and Signature-Agent facts against the primary IETF and Cloudflare sources at build, then verify the result with the Agent-Readiness Audit.

← Agent-Readiness · .md