# Pay, Block or Welcome: AI Access Economics

> AI access economics is the layer of the agentic web where a site owner decides whether to block AI agents, charge them, or welcome them — governed by opt-out tokens, pay-per-crawl and content-licensing standards, each weighed against the traffic and citations the agent returns.

OUTER pillar (Node O2) · block | price | permit

## AI access economics is the permission layer of the agentic web

AI access economics is the layer of the agentic web where a publisher chooses among three options for every AI agent — block it, charge it, or welcome it — and weighs that choice against the traffic, citations and referrals the agent returns. This pillar enumerates every block/price/permit mechanism with who controls it and where it is specified, and presents the block-vs-welcome economics honestly — it is the one node that steelmans blocking. The site's thesis is welcome, but the decision is yours.

- **Block** — refuse access (robots.txt-AI, `noai`/`noimageai`, operator opt-out tokens) → /access-economics/opt-out-tokens
- **Price** — charge for access (Cloudflare pay-per-crawl, RSL usage licenses) → /access-economics/pay-per-crawl
- **Permit** — license access on terms (RSL free / attribution / subscription) → /access-economics/content-licensing-rsl

## Access-control mechanisms divide into block, price and permit types

Every AI access-control mechanism falls into one of three types: **block** (robots.txt-AI, `noai`/`noimageai`, operator opt-out tokens), **price** (Cloudflare pay-per-crawl, RSL usage licenses), or **permit** (RSL free/attribution/subscription). Every mechanism is specified by a Layer-6 licensing protocol (/protocols/licensing).

| Mechanism | Type | Who controls | Specified by | Routes to |
|---|---|---|---|---|
| robots.txt-AI | block | publisher | /protocols/content-signal | opt-out-tokens |
| `noai` / `noimageai` | block | publisher | /protocols/licensing | opt-out-tokens |
| Google-Extended / Applebot-Extended | block (training) | operator | /protocols/licensing | opt-out-tokens |
| Cloudflare pay-per-crawl | price | CDN | /protocols/pay-per-crawl | pay-per-crawl |
| Content-Signal | permit (usage) | publisher | /protocols/content-signal | pay-per-crawl |
| RSL (5 models) | permit / price | publisher | /protocols/rsl | content-licensing-rsl |

No single mechanism does all three jobs, so most sites combine them. Per-bot rationale lives in the registry (/crawlers).

## Opt-out tokens let you decline AI training per operator

Opt-out tokens let you decline AI training on a per-operator basis without leaving search — `Google-Extended` in robots.txt opts a site out of AI training (reported to cover Gemini/Vertex) while keeping Google Search indexing, and `Applebot-Extended` does the same for Apple Intelligence. A training opt-out is not the same as blocking the search/retrieval crawler — block the latter and you forfeit citations. `noai`/`noimageai` are adoption-dependent signals honored at operator discretion; robots.txt `Disallow` is the opt-out of record. Verify exact behaviour against Google Search Central and Apple's Applebot docs at build.

| Token / directive | Operator | Opts out of | Keeps | Where set |
|---|---|---|---|---|
| `Google-Extended` | Google | AI training (Gemini/Vertex, reported) | Google Search indexing | robots.txt |
| `Applebot-Extended` | Apple | Apple Intelligence training | Applebot search | robots.txt |
| `noai` | operator discretion | AI use of page text | (signal only) | meta robots |
| `noimageai` | operator discretion | AI use of page images | (signal only) | meta robots |

```
# robots.txt — opt out of AI training, keep search
User-agent: Google-Extended
Disallow: /

<meta name="robots" content="noai, noimageai">
```

## Pay-per-crawl prices each AI request at the network edge

Pay-per-crawl prices AI access at the network edge: Cloudflare's pay-per-crawl lets a publisher charge AI crawlers a fee per request — returning `HTTP 402 (Payment Required)` to bots that have not paid — so crawling becomes a metered transaction. Content-Signal pairs with it to declare how content may be used after access (search vs train vs AI input), separating the access decision from the usage decision. It is a Layer-6 protocol enforced at the CDN. Pay-per-crawl mechanics, pricing, GA/beta status and any figure must be verified against Cloudflare's primary docs at build.

See the spec: /protocols/pay-per-crawl · Content-Signal: /protocols/content-signal

## RSL licenses content to AI across five business models

RSL (Really Simple Licensing) is a machine-readable content-licensing standard that lets a publisher attach one of five license models to content for AI use — **free**, **attribution** (use it but credit/link), **subscription** (flat-fee access), **pay-per-crawl** (charge per fetch), and **pay-per-inference** (charge each time content is used to generate an answer). Pay-per-crawl prices the fetch; pay-per-inference prices the use in an answer. RSL is a Layer-6 protocol enforced at the edge. Verify the launch date and exact model set against rslstandard.org at build.

| Model | What it permits | How it charges | Best for |
|---|---|---|---|
| Free | open AI use | nothing | reach, citations, visibility |
| Attribution | use with credit + link | nothing (credit required) | brand and referral publishers |
| Subscription | access under a flat fee | flat recurring fee | archives, ongoing access |
| Pay-per-crawl | access priced per fetch | per request | high-value fetch targets |
| Pay-per-inference | use priced per answer | per inference | premium content in AI answers |

See the spec: /protocols/rsl

## Block-vs-welcome economics weigh lost citations against lost control

Whether to block or welcome AI comes down to a single trade-off — blocking gives you control over training and refuses no-referral crawlers, but forfeits the citations and agent referrals that search and retrieval crawlers send back. The right answer depends on which crawler and which content.

**The case for blocking:** training crawlers can ingest content with no referral and no payment; blocking protects proprietary/paywalled content, refuses uncompensated training, and is the only lever a publisher fully controls. For premium archives and licensable datasets, blocking-until-paid is rational — pay-per-crawl and RSL exist to monetize that stance.

**The case for welcoming:** search and retrieval crawlers cite and refer; blocking them removes a site from AI answers entirely, and in the zero-click reality an AI answer may be the only surface a user sees. Welcoming and structuring content captures that citation surface; a transacting agent is a buyer, not a cost.

**The neutral decision:** per-crawler and per-content, not all-or-nothing — block uncompensated training, price high-value retrieval (pay-per-crawl/RSL), welcome citation crawlers. This site's thesis is welcome, but that is a stance, not the only rational choice. Walk it crawler by crawler: /access-economics/should-you-block-ai

| Crawler type | Returns | Default decision |
|---|---|---|
| Uncompensated training crawler | no referral, no payment | block (or price via RSL) |
| High-value retrieval / fetch | fetch of premium content | price (pay-per-crawl / RSL) |
| Citation / answer crawler | citations, referral traffic | welcome |
| Transacting agent | a paying customer | welcome |

## AI access economics compared across Cloudflare, RSL and the opt-out standards

The options differ by who controls them and what they do: opt-out tokens (operator-controlled, block training), Cloudflare pay-per-crawl (CDN-controlled, prices access), Content-Signal (publisher-declared usage), and RSL (publisher-controlled, licenses across five models). No single mechanism does all three jobs, so most sites combine them. Adoption over time: /state-of-the-agentic-web

| Mechanism | Type | Who controls | Enforced by | Best paired with |
|---|---|---|---|---|
| Opt-out tokens | block (training) | operator | operator discretion | a price/permit layer |
| Cloudflare pay-per-crawl | price | CDN | CDN edge (HTTP 402) | Content-Signal |
| Content-Signal | permit (usage) | publisher | CDN edge / robots.txt | pay-per-crawl |
| RSL | permit / price | publisher | CDN edge | pay-per-crawl, attribution |

## AI access economics — frequently asked questions

### What is the difference between pay-per-crawl and pay-per-inference?

Pay-per-crawl prices the fetch — a crawler is charged each time it requests a page. Pay-per-inference prices the use — the publisher is paid each time the content is used to generate an answer. Pay-per-inference aligns payment with the actual value an AI extracts from the content rather than with raw retrieval.

### Can I opt out of AI training without losing search traffic?

Yes. Operator opt-out tokens such as Google-Extended and Applebot-Extended are designed to decline AI training while keeping the search or retrieval crawler welcome, so the site stays indexable and citable. Blocking the search crawler itself, by contrast, removes the site from that engine's answers entirely.

### What is RSL (Really Simple Licensing)?

RSL is a machine-readable content-licensing standard that lets a publisher attach a license to content for AI use across several business models — free, attribution, subscription, pay-per-crawl and pay-per-inference. It expresses terms a machine can read and is enforced at the network edge. Verify the launch date and the exact model set against rslstandard.org.

## From deciding access to implementing and proving it

You have decided whether to block, price, or welcome AI — but how do you implement that directive, and how do you prove an agent is treated the way you intend? Cross into the Layer-6 licensing specs (/protocols/licensing), the per-bot opt-out records (/crawlers), and the audit (/services). AGENTS WELCOME practices what this page documents: it chooses welcome, declares its terms machine-readably, and you can verify the configuration live.