# Pay, Block or Welcome: AI Access Economics

> AI access economics is the layer of the agentic web where a site owner decides whether to block AI agents, charge them, or welcome them — governed by opt-out tokens, pay-per-crawl and content-licensing standards, each weighed against the traffic and citations the agent returns.

OUTER pillar (Node O2) · block | price | permit

## AI access economics is the permission layer of the agentic web

AI access economics is the layer of the agentic web where a publisher chooses among three options for every AI agent — block it, charge it, or welcome it — and weighs that choice against the traffic, citations and referrals the agent returns. This pillar enumerates every block/price/permit mechanism with who controls it and where it is specified, and presents the block-vs-welcome economics honestly — it is the one node that steelmans blocking. The site's thesis is welcome, but the decision is yours.

- **Block** — refuse access (robots.txt-AI, `noai`/`noimageai`, operator opt-out tokens) → /access-economics/opt-out-tokens
- **Price** — charge for access (Cloudflare pay-per-crawl, RSL usage licenses) → /access-economics/pay-per-crawl
- **Permit** — license access on terms (RSL free / attribution / subscription) → /access-economics/content-licensing-rsl

## Access-control mechanisms divide into block, price and permit types

Every AI access-control mechanism falls into one of three types: **block** (robots.txt-AI, `noai`/`noimageai`, operator opt-out tokens), **price** (Cloudflare pay-per-crawl, RSL usage licenses), or **permit** (RSL free/attribution/subscription). Every mechanism is specified by a Layer-6 licensing protocol (/protocols/licensing).

| Mechanism | Type | Who controls | Specified by | Routes to |
|---|---|---|---|---|
| robots.txt-AI | block | publisher | /protocols/licensing/content-signal | opt-out-tokens |
| `noai` / `noimageai` | block | publisher | /protocols/licensing | opt-out-tokens |
| Google-Extended / Applebot-Extended | block (training) | operator | /protocols/licensing | opt-out-tokens |
| Cloudflare pay-per-crawl | price | CDN | /protocols/licensing/pay-per-crawl | pay-per-crawl |
| Content-Signal | permit (usage) | publisher | /protocols/licensing/content-signal | pay-per-crawl |
| RSL (5 models) | permit / price | publisher | /protocols/licensing/rsl | content-licensing-rsl |

No single mechanism does all three jobs, so most sites combine them. Per-bot rationale lives in the registry (/crawlers).

## Opt-out tokens let you decline AI training per operator

Opt-out tokens let you decline AI training on a per-operator basis without leaving search — `Google-Extended` in robots.txt opts a site out of AI training (reported to cover Gemini/Vertex) while keeping Google Search indexing, and `Applebot-Extended` does the same for Apple Intelligence. A training opt-out is not the same as blocking the search/retrieval crawler — block the latter and you forfeit citations. `noai`/`noimageai` are adoption-dependent signals honored at operator discretion; robots.txt `Disallow` is the opt-out of record. Verify exact behaviour against Google Search Central and Apple's Applebot docs at build.

| Token / directive | Operator | Opts out of | Keeps | Where set |
|---|---|---|---|---|
| `Google-Extended` | Google | AI training (Gemini/Vertex, reported) | Google Search indexing | robots.txt |
| `Applebot-Extended` | Apple | Apple Intelligence training | Applebot search | robots.txt |
| `noai` | operator discretion | AI use of page text | (signal only) | meta robots |
| `noimageai` | operator discretion | AI use of page images | (signal only) | meta robots |

```
# robots.txt — opt out of AI training, keep search
User-agent: Google-Extended
Disallow: /

<meta name="robots" content="noai, noimageai">
```

## Pay-per-crawl prices each AI request at the network edge

Pay-per-crawl prices AI access at the network edge: Cloudflare's pay-per-crawl lets a publisher charge AI crawlers a fee per request — returning `HTTP 402 (Payment Required)` to bots that have not paid — so crawling becomes a metered transaction. Content-Signal pairs with it to declare how content may be used after access (search vs train vs AI input), separating the access decision from the usage decision. It is a Layer-6 protocol enforced at the CDN. Pay-per-crawl mechanics, pricing, GA/beta status and any figure must be verified against Cloudflare's primary docs at build.

See the spec: /protocols/licensing/pay-per-crawl · Content-Signal: /protocols/licensing/content-signal

## RSL licenses content to AI across five business models

RSL (Really Simple Licensing) is a machine-readable content-licensing standard that lets a publisher attach one of five license models to content for AI use — **free**, **attribution** (use it but credit/link), **subscription** (flat-fee access), **pay-per-crawl** (charge per fetch), and **pay-per-inference** (charge each time content is used to generate an answer). Pay-per-crawl prices the fetch; pay-per-inference prices the use in an answer. RSL is a Layer-6 protocol enforced at the edge. Verify the launch date and exact model set against rslstandard.org at build.

| Model | What it permits | How it charges | Best for |
|---|---|---|---|
| Free | open AI use | nothing | reach, citations, visibility |
| Attribution | use with credit + link | nothing (credit required) | brand and referral publishers |
| Subscription | access under a flat fee | flat recurring fee | archives, ongoing access |
| Pay-per-crawl | access priced per fetch | per request | high-value fetch targets |
| Pay-per-inference | use priced per answer | per inference | premium content in AI answers |

See the spec: /protocols/licensing/rsl

## Block-vs-welcome economics weigh lost citations against lost control

Whether to block or welcome AI comes down to a single trade-off — blocking gives you control over training and refuses no-referral crawlers, but forfeits the citations and agent referrals that search and retrieval crawlers send back. The right answer depends on which crawler and which content.

**The case for blocking:** training crawlers can ingest content with no referral and no payment; blocking protects proprietary/paywalled content, refuses uncompensated training, and is the only lever a publisher fully controls. For premium archives and licensable datasets, blocking-until-paid is rational — pay-per-crawl and RSL exist to monetize that stance.

**The case for welcoming:** search and retrieval crawlers cite and refer; blocking them removes a site from AI answers entirely, and in the zero-click reality an AI answer may be the only surface a user sees. Welcoming and structuring content captures that citation surface; a transacting agent is a buyer, not a cost.

**The neutral decision:** per-crawler and per-content, not all-or-nothing — block uncompensated training, price high-value retrieval (pay-per-crawl/RSL), welcome citation crawlers. This site's thesis is welcome, but that is a stance, not the only rational choice. Walk it crawler by crawler: /access-economics/should-you-block-ai

| Crawler type | Returns | Default decision |
|---|---|---|
| Uncompensated training crawler | no referral, no payment | block (or price via RSL) |
| High-value retrieval / fetch | fetch of premium content | price (pay-per-crawl / RSL) |
| Citation / answer crawler | citations, referral traffic | welcome |
| Transacting agent | a paying customer | welcome |

## AI access economics compared across Cloudflare, RSL and the opt-out standards

The options differ by who controls them and what they do: opt-out tokens (operator-controlled, block training), Cloudflare pay-per-crawl (CDN-controlled, prices access), Content-Signal (publisher-declared usage), and RSL (publisher-controlled, licenses across five models). No single mechanism does all three jobs, so most sites combine them. Adoption over time: /state-of-the-agentic-web

| Mechanism | Type | Who controls | Enforced by | Best paired with |
|---|---|---|---|---|
| Opt-out tokens | block (training) | operator | operator discretion | a price/permit layer |
| Cloudflare pay-per-crawl | price | CDN | CDN edge (HTTP 402) | Content-Signal |
| Content-Signal | permit (usage) | publisher | CDN edge / robots.txt | pay-per-crawl |
| RSL | permit / price | publisher | CDN edge | pay-per-crawl, attribution |

## From deciding access to implementing and proving it

You have decided whether to block, price, or welcome AI — but how do you implement that directive, and how do you prove an agent is treated the way you intend? Cross into the Layer-6 licensing specs (/protocols/licensing), the per-bot opt-out records (/crawlers), and the audit (/services). AGENTS WELCOME practices what this page documents: it chooses welcome, declares its terms machine-readably, and you can verify the configuration live.