Pay-Per-Crawl: Pricing AI Access at the Edge

Pay-per-crawl prices each AI request at the edge: Cloudflare returns HTTP 402 to unpaid crawlers and settles the fee as merchant of record.

What is pay-per-crawl, and who controls it?

Pay-per-crawl is Cloudflare's CDN-controlled way of pricing AI access at the network edge: the publisher sets one flat, per-request price for the whole site, and the edge answers unpaid AI crawlers with HTTP 402 (Payment Required), so crawling becomes a metered transaction rather than a free take (Cloudflare, 2025). Per crawler the publisher chooses allow (free access), charge (the configured price) or block (no option to pay), and Cloudflare acts as merchant of record, billing the crawler and paying the publisher (Cloudflare, 2025).

The control point is the difference from robots.txt, whose rules "are not a form of access authorization" (IETF, 2022): pay-per-crawl is enforced at the CDN, so a crawler either presents payment or receives the 402, and the charge is tied to a Web Bot Auth signature, so a forged User-Agent cannot pose as a paying crawler (Cloudflare, 2025).

Is pay-per-crawl generally available?

No. As of 7 September 2026, Cloudflare's documentation states that pay-per-crawl "is currently in closed beta", priced per zone, with access by sign-up or through an Enterprise account executive (Cloudflare, 2026). It launched on 1 July 2025 as a private beta, the day Cloudflare moved new domains to a permission-based default for AI crawlers; more than one million customers had already chosen to block them (Cloudflare, 2025). There is no published price list: the launch post shows only placeholder amounts, so any per-request figure quoted elsewhere is a publisher's setting or an estimate, not a Cloudflare rate.

How does the HTTP 402 flow gate an unpaid crawl?

The flow takes two requests. An AI crawler asks for a URL; with no payment on file, the edge answers 402 Payment Required and names the price in a crawler-price header; a crawler that agrees retries with crawler-exact-price and receives 200 OK with the content, which Cloudflare charges and settles (Cloudflare, 2025). A crawler can also state a ceiling up front with crawler-max-price. A 402 is a standard, parseable refusal an agent can react to — unlike a silent block.

GET /article HTTP/1.1
User-Agent: GPTBot

HTTP/1.1 402 Payment Required
crawler-price: USD XX.XX

GET /article HTTP/1.1
User-Agent: GPTBot
crawler-exact-price: USD XX.XX

HTTP/1.1 200 OK
(content served; Cloudflare bills the crawler)

What does Content-Signal add to pay-per-crawl?

Content-Signal adds the usage decision. Pay-per-crawl decides whether a crawler may fetch; one robots.txt line — Content-Signal: search=yes, ai-input=yes, ai-train=no — declares how fetched content may be used afterward: search index, AI input (retrieval-augmented generation, grounding) or AI training. Cloudflare published the Content Signals Policy on 24 September 2025 under CC0 and serves search=yes, ai-train=no by default in its managed robots.txt on more than 3.8 million domains (Cloudflare, 2025). A signal is a preference, not a control: the policy states that "content signals express preferences; they are not technical countermeasures against scraping", and an absent signal neither grants nor restricts (Cloudflare, 2025) — which is why it pairs with a price gate rather than replacing one. This site's robots.txt repeats the line in every user-agent group, because under RFC 9309 a crawler that finds its own group never reads the * group (IETF, 2022).

Pay-per-crawl — frequently asked questions

How does Cloudflare pay-per-crawl charge AI crawlers?

It prices access at the network edge: the publisher sets one flat, per-request price for the whole site, and Cloudflare returns HTTP 402 (Payment Required) with a crawler-price header to AI crawlers that have not paid; a crawler that agrees retries with crawler-exact-price and is served. Cloudflare acts as merchant of record for billing and settlement.

Is Cloudflare pay-per-crawl generally available?

No. As of 7 September 2026, Cloudflare's documentation describes pay-per-crawl as a closed beta, priced per zone and joined by sign-up or through an Enterprise account executive. It launched as a private beta on 1 July 2025.

How much does pay-per-crawl cost per request?

Cloudflare publishes no price list: each publisher sets its own flat per-request price, and Cloudflare's launch post shows only placeholder amounts. Any specific per-request figure quoted elsewhere is a publisher's setting or an estimate, not a Cloudflare rate.

Does pay-per-crawl replace robots.txt or Content-Signal?

No. robots.txt says which crawlers may fetch and Content-Signal says how fetched content may be used, and both are preferences a crawler chooses to honor; pay-per-crawl is enforced at the CDN edge. Sites combine them: a Content-Signal line for usage, a price gate for the fetch.

What happens when a crawler ignores the 402?

It gets no content. The 402 response carries the price but not the page, so a crawler that does not retry with payment is refused — unlike a robots.txt Disallow, which depends on the crawler's compliance.

Related: pay-per-crawl prices each AI request (Layer-6 spec) · x402, AP2, ACP and UCP settle and authorize agent payments · Content-Signal declares how content may be used · pay-per-crawl defined in the Lexicon · each crawler's opt-out mechanism in the registry · back to AI access economics · audit how your site treats agents.

← Access Economics · .md