Pay-Per-Crawl: Pricing AI Access at the Edge
Pay-per-crawl prices each AI request at the edge: Cloudflare returns HTTP 402 to unpaid crawlers and settles the fee as merchant of record.
Pay-per-crawl charges AI crawlers per request
Pay-per-crawl prices AI access at the network edge: a publisher sets a fee per request, and the CDN returns HTTP 402 (Payment Required) to AI crawlers that have not paid — so crawling becomes a metered transaction rather than a free take. Cloudflare's pay-per-crawl is the reference implementation: it sits in front of the origin, identifies known AI crawlers, and gates them on payment instead of an all-or-nothing block. The publisher keeps three outcomes per crawler — allow free, charge, or block — which is what makes pricing a middle path between welcoming everything and refusing everything.
The control point matters: pay-per-crawl is CDN-controlled and enforced at the edge, not a voluntary signal honored at the operator's discretion. A crawler either presents valid payment or receives the 402 — there is no compliance gap of the kind that affects meta-tag opt-outs. Cloudflare acts as merchant of record, handling billing and settlement between the crawler and the publisher so neither side runs its own micropayment rails. The exact pricing model, the list of recognised crawlers, and the GA/beta status are operational details that should be verified against Cloudflare's primary pay-per-crawl docs at build time — no transaction or pricing figure is asserted here.
The HTTP 402 flow gates an unpaid crawl
The mechanism reduces to one status code. An AI crawler requests a URL; the edge checks whether that crawler has a payment arrangement; if not, it answers 402 Payment Required instead of 200 with the content. A paying crawler is charged the per-request fee and served normally. This keeps the access decision (may you fetch this?) cleanly separated from the content, and it is legible to machines — a 402 is a standard, parseable refusal a well-built agent can react to, unlike a silent block.
GET /article HTTP/1.1
User-Agent: GPTBot
HTTP/1.1 402 Payment Required
(crawler has no payment arrangement — fetch is gated until settled)Content-Signal declares how content may be used after access
Pay-per-crawl decides whether a crawler may fetch; the Content-Signal directive declares how the fetched content may be used afterward — for example search indexing versus model training versus use as AI input. Pairing the two separates the access decision from the usage decision: a publisher can price the fetch and still constrain downstream use. Content-Signal is a Layer-6 licensing directive (set in robots.txt or an HTTP header); its exact syntax should be verified against the primary Cloudflare/spec docs at build. Together, pay-per-crawl plus Content-Signal let a site monetise the request and govern the use in one edge configuration.
Related: pay-per-crawl prices each AI request (Layer-6 spec) · Content-Signal declares how content may be used · pay-per-crawl defined in the Lexicon · each crawler's opt-out mechanism in the registry · back to AI access economics · audit how your site treats agents.