Pay, Block or Welcome

What is AI access economics?

AI access economics is the layer of the agentic web where a publisher chooses one of three options for every AI agent — block it, charge it, or welcome it — and weighs that choice against the traffic, citations and referrals the agent returns. It is the money-and-permission layer that decides who pays, who is blocked, and who is welcomed, one level above the protocols that enforce each decision.

The bargain is lopsided. According to Cloudflare (2025), 80 % of the AI crawling it observed in the 12 months before its August 2025 report served model training, 18 % served search and 2 % served user-initiated actions — and only the last two send anyone back. The site's thesis is welcome; the decision is yours, and it turns on which crawler and which content. The site's own AI Crawler Registry records the documented opt-out mechanism and a block-versus-allow recommendation for 41 crawlers and policy tokens (AGENTS WELCOME, updated 2026-07-06).

• Block — refuse access (robots.txt-AI, noai/noimageai, operator opt-out tokens). Routes to operator opt-out tokens like Google-Extended.
• Price — charge for access (Cloudflare pay-per-crawl, RSL usage licenses). Routes to the pay-per-crawl mechanism explained in full.
• Permit — license access on terms (RSL free / attribution / subscription). Routes to RSL content licensing and its five models.

Which mechanisms block, price or permit AI access?

Every AI access-control mechanism is one of three types — block (refuse access), price (charge for access) or permit (license access on terms) — and a different party sets and honors each one; every mechanism here is specified by a Layer-6 licensing protocol in the Protocol Atlas. The table lists all six with controller, specification and sub-topic; RSL alone spans two types.

Per-bot rationale — each crawler's opt-out mechanism and block-vs-allow rationale — lives in the registry.

How do opt-out tokens decline AI training without leaving search?

Opt-out tokens decline AI training per operator while that operator's search crawler keeps indexing and citing you. As of 14 July 2026, the last update of Google's crawler documentation, Google-Extended and other directives defined in the Lexicon work like this: Google-Extended is a standalone robots.txt product token, and disallowing it withholds crawled content from training future Gemini models and from grounding them at prompt time, while Google states it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal" (Google, 2026). It has no user-agent string of its own. Applebot-Extended "does not crawl webpages", opts content out of training the foundation models behind Apple Intelligence, and pages that disallow it "can still be included in search results" (Apple, 2026). The load-bearing distinction: a training opt-out is not a search block — block the search crawler and you forfeit the citations that search and AI answers send back.

The noai and noimageai meta directives signal at page level that text and images should not be used for AI; operators honor them at their discretion. A per-crawler Disallow in robots.txt remains the opt-out of record — robots.txt-AI blocks AI crawlers per user-agent at the spec level — yet RFC 9309 states that its rules "are not a form of access authorization" (IETF, 2022), which is why the price layer below exists.

# robots.txt — opt out of AI training, keep search
User-agent: Google-Extended
Disallow: /

<!-- content-level refusal signal -->
<meta name="robots" content="noai, noimageai">

How does pay-per-crawl price each AI request at the network edge?

Pay-per-crawl prices AI access at the network edge: Cloudflare's pay-per-crawl lets a publisher set one flat, per-request price for the whole site and answer unpaid AI crawlers with HTTP 402 (Payment Required), so crawling becomes a metered transaction (Cloudflare, 2025). It launched on 1 July 2025 as a private beta; Cloudflare's documentation (last updated 28 July 2026) still described a closed beta, priced per zone, when this page was verified on 7 September 2026 (Cloudflare, 2026). Per crawler the publisher chooses allow, charge or block; the 402 carries a crawler-price header, an agreeing crawler retries with crawler-exact-price, and Cloudflare acts as merchant of record (Cloudflare, 2025). It is a Layer-6 licensing decision enforced at the CDN — see pay-per-crawl defined in the Agentic Web Lexicon and pay-per-crawl prices each AI request (Layer-6 spec).

Pricing pairs with a usage declaration. Content-Signal declares how content may be used after access with one robots.txt line — Content-Signal: search=yes, ai-input=yes, ai-train=no — covering a search index, AI input such as retrieval-augmented generation or grounding, and AI training. Cloudflare published the policy on 24 September 2025 under CC0 and announced search=yes, ai-train=no as the default for the managed robots.txt it serves on more than 3.8 million domains, while stating that "content signals express preferences; they are not technical countermeasures against scraping" (Cloudflare, 2025). That separates the access decision (may the bot fetch this?) from the usage decision (what may it do with what it fetched?).

# Pay-per-crawl flow (conceptual)
AI crawler  ──GET /article──▶  Cloudflare edge
              ◀─402 Payment Required─  (no payment on file; crawler-price header)
AI crawler  ──retry, crawler-exact-price──▶   Cloudflare edge
              ◀─200 OK + content─  (metered, settled)

Which license models does RSL offer for AI use?

RSL (Really Simple Licensing) launched on 10 September 2025 with five license models for AI use — free, attribution (use it, but credit and link the source), subscription (flat-fee access), pay-per-crawl (charge per fetch) and pay-per-inference (charge each time the content is used to generate an answer) — backed at launch by Reddit, Yahoo, O'Reilly Media and Medium, among others (RSL Collective, 2025). RSL 1.0 became a Recommendation on 10 December 2025, endorsed by more than 1,500 organizations, with Cloudflare, Akamai, Creative Commons and IAB Tech Lab among its supporters (RSL Collective, 2025). RSL defined in the Lexicon gives the term; the standard itself is recorded at RSL licenses content to AI across five models.

A machine reads the license before it acts: robots.txt names the document with a License: directive, and the XML declares per usage type — search, ai-input, ai-index, ai-train, ai-all or all — what is permitted and under which payment type. The two usage-priced models meter different moments: the crawl type is "payment every time the content is crawled", the use type "payment each time the content contributes to an AI-generated output" (RSL Collective, 2025) — the latter aligns payment with the value an AI extracts. Enforcement sits outside the standard, in the CDN or paywall a publisher pairs it with. The table lists the five launch models with what each permits and charges.

This site practices it: its RSL license at /license.xml permits search, AI input and summarization free and licenses training use at a demo price, and every HTML response advertises it in an RFC 8288 Link header with rel="license" — the terms travel with the content, not in a contract a bot cannot parse.

What does blocking AI forfeit, and what does it protect?

Blocking AI trades citations for control: it stops training and turns away crawlers that never refer anyone, but it also removes you from the search and answer surfaces that do — so the right answer depends on which crawler and which content. You can walk the block-vs-welcome decision crawler by crawler in the dedicated walkthrough.

The case for blocking AI: control, training refusal, no-referral crawlers

Training crawlers take without giving back. According to Cloudflare (2025), in July 2025 Anthropic's crawlers made 38,065.7 requests for every referral they sent back, OpenAI's 1,091.4 and Perplexity's 194.8, against 5.4 for Google. Blocking protects proprietary and paywalled content, refuses uncompensated training, and is the one lever a publisher fully controls. For premium archives, licensable datasets and original reporting, blocking-until-paid is the rational position, and pay-per-crawl and RSL exist to monetize that stance; more than one million Cloudflare customers had chosen to block AI crawlers by July 2025 (Cloudflare, 2025). Per-bot rationale lives in each crawler's opt-out mechanism and block-vs-allow rationale.

The case for welcoming AI: citations, agent referrals, the zero-click reality

Search and retrieval crawlers cite and refer, and the AI answer is increasingly the only surface a user sees. According to Pew Research Center (2025), U.S. Google users clicked a traditional result in 8 % of visits when an AI summary appeared versus 15 % when none did, and ended their session on 26 % versus 16 % of such pages (68,879 searches, March 2025). Blocking the crawlers that feed those answers removes a site from them entirely; welcoming them captures that citation surface, and an agent that can transact is a buyer, not a cost.

A neutral decision: block training, price retrieval, welcome citation

The defensible synthesis is per-crawler and per-content, not all-or-nothing: block uncompensated training, price high-value retrieval (pay-per-crawl or RSL), welcome citation crawlers. The table gives the default decision per crawler type.

How do Cloudflare pay-per-crawl, RSL and the opt-out tokens compare?

The access-economics options differ by who controls them and what they do: opt-out tokens (operator-controlled, decline training), Cloudflare pay-per-crawl (CDN-controlled, prices access; closed beta as of September 2026), Content-Signal (a publisher-declared preference, Cloudflare's default on the managed robots.txt of more than 3.8 million domains) and RSL (publisher-controlled, five models, 1,500+ endorsing organizations) — none does all three jobs, so most sites combine them (Cloudflare, 2025; RSL Collective, 2025). IAB Tech Lab's LLM Content Ingest API initiative adds a further price-layer proposal — a content-ingest API with cost-per-crawl and query-based monetization, continued in its Content Monetization Protocols working group (IAB Tech Lab, 2025). How adoption of each access standard measured over time is tracked in the adoption report, where GPTBot alone accounted for 11.48 % of observed AI-crawler requests in May 2026 (Cloudflare Radar, as recorded by AGENTS WELCOME, 2026).

Each mechanism owns a different job and controller, so the practical answer is a stack: decline uncompensated training at the operator layer, price high-value retrieval at the CDN, license the rest with RSL.

AI access economics — frequently asked questions

What is the difference between pay-per-crawl and pay-per-inference?

Pay-per-crawl prices the fetch — a crawler is charged each time it requests a page; RSL 1.0 calls this the crawl payment type, 'payment every time the content is crawled'. Pay-per-inference prices the use — RSL 1.0's use type is 'payment each time the content contributes to an AI-generated output' (RSL Collective, 2025). Pay-per-inference aligns payment with the value an AI extracts rather than with raw retrieval.

Can I opt out of AI training without losing search traffic?

Yes. Google states that Google-Extended does not affect a site's inclusion or ranking in Google Search (Google, 2026), Apple states that pages disallowing Applebot-Extended can still be included in search results (Apple, 2026), and OpenAI documents GPTBot (training) and OAI-SearchBot (search) as independent settings (OpenAI, 2026). Blocking the search crawler itself removes the site from that engine's answers.

What is RSL (Really Simple Licensing)?

RSL is an open, machine-readable content-licensing standard: a robots.txt License directive points at an XML document that states, per usage type, what AI may do with the content and what it costs. It launched on 10 September 2025 with five models — free, attribution, subscription, pay-per-crawl and pay-per-inference — and RSL 1.0 became a Recommendation on 10 December 2025 (RSL Collective, 2025). Enforcement sits outside the standard.

How do you implement and prove the access decision?

You have decided whether to block, price, or welcome AI on your site — but how do you implement that directive, and how do you prove an agent is treated the way you intend? Every mechanism on this page is specified by a Layer-6 licensing protocol; the per-crawler records live in the registry's opt-out mechanism and block-vs-allow rationale; and you can audit how your site treats agents and certify it.

AGENTS WELCOME practices what this page documents: it chooses welcome, declares its terms machine-readably, and you can verify the configuration live — pay-per-crawl prices each AI request, RSL licenses content across five models, Content-Signal declares how content may be used, robots.txt-AI blocks AI crawlers per user-agent, and you can walk the block-vs-welcome decision crawler by crawler before you choose.