# Pay-Per-Crawl: Pricing AI Access at the Edge

Published 2026-06-17 · Updated 2026-09-07 · Author: Özden Erdinc (https://erdinc.ai) · Editorial guidelines: /about#editorial-guidelines

> Pay-per-crawl prices each AI request at the edge: Cloudflare returns HTTP 402 to unpaid crawlers and settles the fee as merchant of record.

## What is pay-per-crawl, and who controls it?

Pay-per-crawl is Cloudflare's CDN-controlled way of pricing AI access at the network edge: the publisher sets one flat, per-request price for the whole site, and the edge answers unpaid AI crawlers with **HTTP 402 (Payment Required)**, so crawling becomes a metered transaction rather than a free take (Cloudflare, 2025). Per crawler the publisher chooses *allow* (free access), *charge* (the configured price) or *block* (no option to pay), and Cloudflare acts as **merchant of record**, billing the crawler and paying the publisher (Cloudflare, 2025).

The control point is the difference from robots.txt, whose rules "are not a form of access authorization" (IETF, 2022): pay-per-crawl is enforced at the CDN, so a crawler either presents payment or receives the 402, and the charge is tied to a Web Bot Auth signature, so a forged User-Agent cannot pose as a paying crawler (Cloudflare, 2025).

## Is pay-per-crawl generally available?

No. As of 7 September 2026, Cloudflare's documentation states that pay-per-crawl "is currently in closed beta", priced per zone, with access by sign-up or through an Enterprise account executive (Cloudflare, 2026). It launched on 1 July 2025 as a private beta, the day Cloudflare moved new domains to a permission-based default for AI crawlers; more than one million customers had already chosen to block them (Cloudflare, 2025). There is no published price list: the launch post shows only placeholder amounts, so any per-request figure quoted elsewhere is a publisher's setting or an estimate, not a Cloudflare rate.

## How does the HTTP 402 flow gate an unpaid crawl?

The flow takes two requests. An AI crawler asks for a URL; with no payment on file, the edge answers **402 Payment Required** and names the price in a `crawler-price` header; a crawler that agrees retries with `crawler-exact-price` and receives 200 OK with the content, which Cloudflare charges and settles (Cloudflare, 2025). A crawler can also state a ceiling up front with `crawler-max-price`. A 402 is a standard, parseable refusal an agent can react to — unlike a silent block.

```
GET /article HTTP/1.1
User-Agent: GPTBot

HTTP/1.1 402 Payment Required
crawler-price: USD XX.XX

GET /article HTTP/1.1
User-Agent: GPTBot
crawler-exact-price: USD XX.XX

HTTP/1.1 200 OK
(content served; Cloudflare bills the crawler)
```

## What does Content-Signal add to pay-per-crawl?

Content-Signal adds the usage decision. Pay-per-crawl decides *whether* a crawler may fetch; one robots.txt line — `Content-Signal: search=yes, ai-input=yes, ai-train=no` — declares how fetched content may be used afterward: search index, AI input (retrieval-augmented generation, grounding) or AI training. Cloudflare published the Content Signals Policy on 24 September 2025 under CC0 and serves `search=yes, ai-train=no` by default in its managed robots.txt on more than 3.8 million domains (Cloudflare, 2025). A signal is a preference, not a control: the policy states that "content signals express preferences; they are not technical countermeasures against scraping", and an absent signal neither grants nor restricts (Cloudflare, 2025) — which is why it pairs with a price gate rather than replacing one. This site's robots.txt repeats the line in every user-agent group, because under RFC 9309 a crawler that finds its own group never reads the `*` group (IETF, 2022).

## Pay-per-crawl — frequently asked questions

### How does Cloudflare pay-per-crawl charge AI crawlers?

It prices access at the network edge: the publisher sets one flat, per-request price for the whole site, and Cloudflare returns HTTP 402 (Payment Required) with a crawler-price header to AI crawlers that have not paid; a crawler that agrees retries with crawler-exact-price and is served. Cloudflare acts as merchant of record for billing and settlement.

### Is Cloudflare pay-per-crawl generally available?

No. As of 7 September 2026, Cloudflare's documentation describes pay-per-crawl as a closed beta, priced per zone and joined by sign-up or through an Enterprise account executive. It launched as a private beta on 1 July 2025.

### How much does pay-per-crawl cost per request?

Cloudflare publishes no price list: each publisher sets its own flat per-request price, and Cloudflare's launch post shows only placeholder amounts. Any specific per-request figure quoted elsewhere is a publisher's setting or an estimate, not a Cloudflare rate.

### Does pay-per-crawl replace robots.txt or Content-Signal?

No. robots.txt says which crawlers may fetch and Content-Signal says how fetched content may be used, and both are preferences a crawler chooses to honor; pay-per-crawl is enforced at the CDN edge. Sites combine them: a Content-Signal line for usage, a price gate for the fetch.

### What happens when a crawler ignores the 402?

It gets no content. The 402 response carries the price but not the page, so a crawler that does not retry with payment is refused — unlike a robots.txt Disallow, which depends on the crawler's compliance.

## Sources

1. Cloudflare: Introducing pay per crawl (1 July 2025), 2025. https://blog.cloudflare.com/introducing-pay-per-crawl/
2. Cloudflare: What is Pay Per Crawl? (AI Crawl Control docs, read 2026-09-07), 2026. https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/
3. Cloudflare: Cloudflare just changed how AI crawlers scrape the internet-at-large (press release, 1 July 2025), 2025. https://www.cloudflare.com/press-releases/2025/cloudflare-just-changed-how-ai-crawlers-scrape-the-internet-at-large/
4. Cloudflare: Content Signals Policy (announcement, 24 September 2025), 2025. https://blog.cloudflare.com/content-signals-policy/
5. IETF: RFC 9309: Robots Exclusion Protocol, 2022. https://www.rfc-editor.org/rfc/rfc9309.html
6. AGENTS WELCOME: This site's robots.txt — a Content-Signal line in every user-agent group, 2026. /robots.txt

Related: [pay-per-crawl Layer-6 spec](/protocols/pay-per-crawl) · [how x402, AP2, ACP and UCP settle and authorize agent payments](/commerce/x402-vs-ap2-vs-acp-vs-ucp) · [Content-Signal spec](/protocols/content-signal) · [pay-per-crawl in the Lexicon](/glossary/pay-per-crawl) · [crawler registry](/crawlers) · [AI access economics](/access-economics) · [audit your site](/services)

