Pay, Block or Welcome

AI access economics is the permission layer of the agentic web

AI access economics is the layer of the agentic web where a publisher chooses among three options for every AI agent — block it, charge it, or welcome it — and weighs that choice against the traffic, citations and referrals the agent returns. It is the money-and-permission layer that decides who pays, who is blocked, and who is welcomed, sitting one level above the protocols that actually enforce each decision.

This pillar does two jobs. First, it enumerates every block, price and permit mechanism with who controls it and where it is specified. Second, it presents the block-versus-welcome economics honestly: this is the one node on the site that steelmans the case for blocking before stating its own stance. The site's thesis is welcome — but the decision is yours, and it depends on which crawler and which content. The bargain underneath is simple to state: an AI crawler takes your content; in return it may give back a citation, a referral, or a paying transaction — or it may give back nothing at all, which is exactly when blocking earns its keep.

• Block — refuse access (robots.txt-AI, noai/noimageai, operator opt-out tokens). Routes to operator opt-out tokens like Google-Extended.
• Price — charge for access (Cloudflare pay-per-crawl, RSL usage licenses). Routes to the pay-per-crawl mechanism explained in full.
• Permit — license access on terms (RSL free / attribution / subscription). Routes to RSL content licensing and its five models.

Access-control mechanisms divide into block, price and permit types

Every AI access-control mechanism falls into one of three types: block (refuse access — robots.txt-AI, noai/noimageai meta directives, operator opt-out tokens), price (charge for access — Cloudflare pay-per-crawl, RSL usage licenses), or permit (license access on terms — RSL free, attribution and subscription). Each mechanism is set and enforced by a different party, and every mechanism here is specified by a Layer-6 licensing protocol in the Protocol Atlas.

No single mechanism does all three jobs, which is why most sites combine a block directive (for uncompensated training), a price gate (for high-value retrieval) and a permit/license (for everything they are willing to share on terms). Per-bot block-versus-allow rationale — each crawler's opt-out mechanism and block-vs-allow rationale — lives in the registry.

Opt-out tokens let you decline AI training per operator

Opt-out tokens let you decline AI training on a per-operator basis without leaving search. Google-Extended and other directives defined in the Lexicon work in robots.txt: Google-Extended opts a site out of AI training (reported to cover Gemini and Vertex AI generative models) while keeping Google Search indexing, and Applebot-Extended does the same for Apple Intelligence. The load-bearing distinction is this: a training opt-out is not the same as blocking the search or retrieval crawler — block the latter and you forfeit the citations that search and AI answers send back.

The noai and noimageai meta directives signal at the content level that a page's text and images should not be used for AI; they are adoption-dependent — honored at each operator's discretion rather than guaranteed. The robots.txt per-crawler Disallow remains the opt-out of record, and robots.txt-AI blocks AI crawlers per user-agent at the spec level. The exact token behaviour and adoption status should be verified against each operator's primary docs at build (Google Search Central for Google-Extended, Apple's Applebot docs for Applebot-Extended) rather than asserted from memory.

# robots.txt — opt out of AI training, keep search
User-agent: Google-Extended
Disallow: /

<!-- content-level refusal signal -->
<meta name="robots" content="noai, noimageai">

Pay-per-crawl prices each AI request at the network edge

Pay-per-crawl prices AI access at the network edge: Cloudflare's pay-per-crawl lets a publisher charge AI crawlers a fee per request — returning HTTP 402 (Payment Required) to bots that have not paid — so that crawling becomes a metered transaction rather than a free take. It is a Layer-6 licensing decision enforced at the CDN, and pay-per-crawl defined in the Agentic Web Lexicon sits alongside the full specification: pay-per-crawl prices each AI request (Layer-6 spec).

Pricing is paired with a usage declaration. Content-Signal declares how content may be used after access — for example whether content is available for search, for training, or as AI input — which separates the access decision (may the bot fetch this?) from the usage decision (what may it do with what it fetched?). The pay-per-crawl mechanics, pricing model, general-availability or beta status, and any transaction or adoption figure are not asserted here; they must be verified against Cloudflare's primary pay-per-crawl announcement and docs at build.

# Pay-per-crawl flow (conceptual)
AI crawler  ──GET /article──▶  Cloudflare edge
              ◀─402 Payment Required─  (no payment on file)
AI crawler  ──pay + retry──▶   Cloudflare edge
              ◀─200 OK + content─  (metered, settled)

RSL licenses content to AI across five business models

RSL (Really Simple Licensing) is a machine-readable content-licensing standard that lets a publisher attach one of five license models to content for AI use — free, attribution (use it but credit and link the source), subscription (flat-fee access), pay-per-crawl (charge per fetch), and pay-per-inference (charge each time the content is used to generate an answer). RSL defined in the Lexicon gives the term; the standard itself is recorded at RSL licenses content to AI across five models.

The two usage-priced models differ in what they meter: pay-per-crawl prices the fetch, while pay-per-inference prices the use in an answer — the latter aligns the publisher's payment with the actual value an AI extracts. RSL is a Layer-6 licensing protocol, expressed in a way a machine can read and enforced at the network edge (for example via Cloudflare). The RSL launch date, the exact set and names of license models, and any adopter figure should be verified against the primary rslstandard.org spec at build rather than asserted from an internal source.

Because the license is machine-readable, an agent can read the terms before it acts — which is the point of the agentic web's permission layer: the terms travel with the content, not in a contract a bot cannot parse.

Block-vs-welcome economics weigh lost citations against lost control

Whether to block or welcome AI comes down to a single trade-off — blocking gives you control over training and refuses no-referral crawlers, but it forfeits the citations and agent referrals that search and retrieval crawlers send back, so the right answer depends on which crawler and which content. The two cases below are both stated at their strongest; you can walk the block-vs-welcome decision crawler by crawler in the dedicated walkthrough.

The case for blocking AI: control, training refusal, no-referral crawlers

Training crawlers can ingest content with no referral and no payment. Blocking protects proprietary and paywalled content, refuses uncompensated training, and is the only lever a publisher fully controls. For some publishers — premium archives, licensable datasets, original reporting — blocking-until-paid is the rational position, and pay-per-crawl and RSL exist precisely to monetize that stance. This is not paranoia; it is a publisher declining to subsidize a model that returns nothing, and per-bot block-versus-allow rationale lives in each crawler's opt-out mechanism and block-vs-allow rationale.

The case for welcoming AI: citations, agent referrals, the zero-click reality

Search and retrieval crawlers cite and refer. Blocking them removes a site from AI answers entirely — and in the zero-click reality, an AI answer may be the only surface a user ever sees. Welcoming those crawlers and structuring content for them captures that citation surface, and an agent that can transact is a buyer, not a cost. For most publishers the citation a retrieval crawler returns is worth more than the control a blanket block preserves.

A neutral decision: block training, price retrieval, welcome citation

The defensible synthesis is per-crawler and per-content, not all-or-nothing: block uncompensated training, price high-value retrieval (pay-per-crawl or RSL), welcome citation crawlers. This site's thesis is welcome — but that is a stance, not the only rational choice, and the decision is yours, turning on content type, referral value and licensing leverage.

AI access economics compared across Cloudflare, RSL and the opt-out standards

The access-economics options differ by who controls them and what they do: opt-out tokens (operator-controlled, block training), Cloudflare pay-per-crawl (CDN-controlled, prices access), Content-Signal (publisher-declared usage terms), and RSL (publisher-controlled, licenses across five models) — no single mechanism does all three jobs, so most sites combine them. How adoption of each access standard measured over time is tracked in the adoption report.

Because each mechanism owns a different job and a different controller, the practical answer is a stack, not a single switch: a publisher typically declines uncompensated training at the operator layer, prices high-value retrieval at the CDN, and licenses the rest with RSL — then hands the implementation to the protocol specs.

From deciding access to implementing and proving it

You have decided whether to block, price, or welcome AI on your site — but how do you actually implement that directive, and how do you prove an agent is being treated the way you intend? That question cannot be answered with more access-economics facts; it is answered by the licensing-protocol specs, the per-bot opt-out records, and an audit. Every mechanism on this page is specified by a Layer-6 licensing protocol; the per-crawler records live in the registry's opt-out mechanism and block-vs-allow rationale; and you can audit how your site treats agents and certify it.

AGENTS WELCOME practices what this page documents: it chooses welcome, declares its terms machine-readably, and you can verify the configuration live — pay-per-crawl prices each AI request, RSL licenses content across five models, Content-Signal declares how content may be used, robots.txt-AI blocks AI crawlers per user-agent, and you can walk the block-vs-welcome decision crawler by crawler before you choose.