# Pay Per Crawl

> Cloudflare's HTTP 402-based mechanism that lets publishers charge AI crawlers per request — allow free, set a price, or block — with Cloudflare as merchant of record.

_The Agent Protocol Atlas · /protocols/pay-per-crawl · [JSON](/api/protocols/pay-per-crawl) · [all The Agent Protocol Atlas](/protocols)_

- **name:** Pay Per Crawl
- **full_name:** Cloudflare Pay Per Crawl
- **layer:** licensing
- **creator:** Cloudflare
- **status:** live (beta, 2025)
- **year:** 2025
- **one_liner:** Cloudflare's HTTP 402-based mechanism that lets publishers charge AI crawlers per request — allow free, set a price, or block — with Cloudflare as merchant of record.
- **spec_url:** https://blog.cloudflare.com/introducing-pay-per-crawl/
- **snippet:**

```
HTTP/1.1 402 Payment Required
crawler-price: USD 0.01   →   crawler retries with: crawler-exact-price: USD 0.01
```
- **abbreviation:** Pay Per Crawl
- **also_known_as:** Cloudflare Pay Per Crawl, pay-per-crawl
- **canonical_spec_url:** https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/
- **entity_uri:** https://blog.cloudflare.com/introducing-pay-per-crawl/
- **taxonomy_layer:** licensing
- **sub_layer:** per-crawl-paywall
- **protocol_type:** settlement
- **central_problem:** Lets a publisher charge AI crawlers for each request to their content — flipping unpaid scraping into a metered, monetizable transaction over HTTP 402.
- **maintainer:** Cloudflare (Pay Per Crawl / AI Crawl Control)
- **governance_body:** vendor (Cloudflare)
- **license:** — (verify-against-primary-at-build)
- **maturity_tag:** emerging
- **current_spec_version:** — (verify-against-primary-at-build)
- **spec_date:** — (verify-against-primary-at-build)
- **launch_date:** 2025-07-01
- **last_verified:** 2026-06-15
- **transport:** HTTP 402 Payment Required + crawler-price / crawler-exact-price headers (Cloudflare network)
- **core_mechanism:** When a configured AI crawler requests content, Cloudflare can return HTTP 402 Payment Required with a crawler-price header; a crawler that agrees retries with a crawler-exact-price header, and Cloudflare — acting as merchant of record — aggregates billing, charges the crawler, and pays the publisher. Publishers can allow free, charge a flat per-request rate, or block.
- **discovery_endpoint:** HTTP 402 response with crawler-price header (per-request; Cloudflare-managed)
- **settlement_type:** credits (Cloudflare as merchant of record; per-request billing)
- **adoption_metric:** — (verify-against-primary-at-build)
- **notable_adopters:** {"value":"Cloudflare (operator / merchant of record)","source":"https://blog.cloudflare.com/introducing-pay-per-crawl/"}
- **relationships:** {"predicate":"verified_by","target":"web-bot-auth","note":"Pay Per Crawl -verified_by-> Web Bot Auth (research §2 seed triple, inverse of 'Web Bot Auth verifies pay-per-crawl'): charging the right crawler depends on cryptographically verifying which agent is real."}, {"predicate":"competes_with","target":"rsl","note":"Pay Per Crawl and RSL both price/gate AI access in the licensing layer; RSL is an open licensing spec, Pay Per Crawl is a Cloudflare-operated paywall."}, {"predicate":"built_on","target":"x402","note":"Pay Per Crawl uses the HTTP 402 Payment Required pattern that x402 also revives."}
- **ideal_use_case:** A publisher behind Cloudflare who wants to monetize AI crawling per request without building payment infrastructure.
- **when_to_use:** When you are on Cloudflare and want to charge AI crawlers per request (or block them) with Cloudflare handling billing and settlement.
- **when_not_to_use:** When you want an open, vendor-neutral licensing declaration (use RSL) or you are not behind Cloudflare's network.
- **code_example:** # Cloudflare returns to an AI crawler:
HTTP/1.1 402 Payment Required
crawler-price: USD 0.01

# Crawler agrees and retries:
GET /article HTTP/1.1
crawler-exact-price: USD 0.01
- **source:** Launch (private beta, July 1 2025), HTTP 402 + crawler-price/crawler-exact-price headers, Cloudflare as merchant of record: https://blog.cloudflare.com/introducing-pay-per-crawl/ . Seed triple (Web Bot Auth verifies pay-per-crawl): research §2.
- **agent_readiness_link:** access-economics
- **layer_legacy:** content