Access Control — Letting the Right Agents In

The agent-readiness dimension that lets the right agents in and verifies them — Web Bot Auth, robots-for-AI, opt-out tokens and pay-per-crawl signals.

What does the access control dimension mean?

Access control is the agent-readiness dimension that decides which agents may use your site and proves who is calling. Its two jobs: verify that a request claiming to be a known agent is genuine, and declare your policy for automated access — allow, deny, license or charge. Discoverability and content make a site readable; access control makes it governable.

Which signals and standards does access control cover?

Four signal families:

• Web Bot Auth — RFC 9421 (IETF, 2024) defines HTTP Message Signatures, EdDSA over edwards25519 included; the IETF webbotauth working group's draft (Meunier and Major, 2026) adds the Signature-Agent header for in-band key discovery and a JWKS directory at /.well-known/http-message-signatures-directory. Cloudflare's verifier accepts Ed25519 keys only and requires that directory over HTTPS (Cloudflare, 2026).
• robots.txt for AI — RFC 9309 (IETF, 2022) matches a crawler to the group naming its product token, so GPTBot, ClaudeBot and PerplexityBot each need their own group. Google-Extended has no user agent of its own; it only controls whether crawled content trains or grounds Gemini (Google, 2026).
• Content Signals — three robots.txt preferences, search, ai-input and ai-train, each yes or no; Cloudflare launched the policy on 24 September 2025 and applies search=yes, ai-train=no by default to more than 3.8 million domains on its managed robots.txt (Cloudflare, 2025).
• Pay-per-crawl and RSL — Cloudflare's pay-per-crawl, in private beta since 1 July 2025, answers an unpaid crawler with HTTP 402 and crawler-price headers, with Cloudflare as merchant of record (Cloudflare, 2025); RSL 1.0 (RSL Collective, 10 December 2025) adds machine-readable license terms alongside robots.txt, endorsed by more than 1,500 organizations.

How does the Agent-Readiness Audit score access control?

Two of the Audit's 18 checks touch access control, worth 12 of 100 points: robots-ai-aware (7, Discovery) passes when /robots.txt names at least one AI crawler such as GPTBot, ClaudeBot or PerplexityBot, and web-bot-auth (5, Trust) passes when /.well-known/agents.json declares an identity block advertising Web Bot Auth (Agents Welcome, 2026). No check scores Content Signals, pay-per-crawl or RSL yet. The Audit reads declarations; this site's /api/whoami does the cryptography live: it verifies an Ed25519 HTTP message signature and reports signature.verified, trusted and keyid against /data/trusted-keys.json. As of 6 July 2026 this site's Crawler Registry holds 41 records — 13 user-action fetchers, 10 search-index crawlers, 9 training crawlers, 4 agentic browsers, 3 data providers and 2 opt-out tokens — of which 16 publish an IP-range file, and Web Bot Auth support is unconfirmed for all 41.

Access control — frequently asked questions

Is a user-agent string proof of an agent's identity?

No. Any client can send GPTBot as its user agent. Proof comes from the operator's published IP ranges, reverse DNS, or a Web Bot Auth signature under RFC 9421 that validates against the operator's key directory.

Do user-triggered fetchers obey robots.txt?

Often not. OpenAI states that robots.txt rules may not apply to ChatGPT-User, and Perplexity states that Perplexity-User generally ignores robots.txt rules, because a person requested the page. Crawlers such as GPTBot and PerplexityBot state that they honor it.

What does Content-Signal add to robots.txt?

A preference line per group with three signals, search, ai-input and ai-train, each set to yes or no. robots.txt stays the allow-or-disallow layer; Content Signals say how fetched content may be used, and a signal placed only in the * group never reaches a crawler that has its own group.

Related: the Web Bot Auth spec · pay-per-crawl licensing · implement Web Bot Auth · verify crawlers · pay, block or welcome AI · audit your site

← Agent-Readiness · .md