Pay Per Crawl
Cloudflare's HTTP 402-based mechanism that lets publishers charge AI crawlers per request — allow free, set a price, or block — with Cloudflare as merchant of record.
- name
- Pay Per Crawl
- full_name
- Cloudflare Pay Per Crawl
- layer
- licensing
- creator
- Cloudflare
- status
- live (beta, 2025)
- year
- 2025
- one_liner
- Cloudflare's HTTP 402-based mechanism that lets publishers charge AI crawlers per request — allow free, set a price, or block — with Cloudflare as merchant of record.
- spec_url
- https://blog.cloudflare.com/introducing-pay-per-crawl/
- snippet
HTTP/1.1 402 Payment Required crawler-price: USD 0.01 → crawler retries with: crawler-exact-price: USD 0.01- abbreviation
- Pay Per Crawl
- also_known_as
Cloudflare Pay Per Crawlpay-per-crawl- canonical_spec_url
- https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/
- entity_uri
- https://blog.cloudflare.com/introducing-pay-per-crawl/
- taxonomy_layer
- licensing
- sub_layer
- per-crawl-paywall
- protocol_type
- settlement
- central_problem
- Lets a publisher charge AI crawlers for each request to their content — flipping unpaid scraping into a metered, monetizable transaction over HTTP 402.
- maintainer
- Cloudflare (Pay Per Crawl / AI Crawl Control)
- governance_body
- vendor (Cloudflare)
- license
- — verify-against-primary-at-build ↗ https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/
- maturity_tag
- emerging
- current_spec_version
- — verify-against-primary-at-build ↗ https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/
- spec_date
- — verify-against-primary-at-build ↗ https://blog.cloudflare.com/introducing-pay-per-crawl/
- launch_date
- 2025-07-01
- last_verified
- 2026-06-15
- transport
- HTTP 402 Payment Required + crawler-price / crawler-exact-price headers (Cloudflare network)
- core_mechanism
- When a configured AI crawler requests content, Cloudflare can return HTTP 402 Payment Required with a crawler-price header; a crawler that agrees retries with a crawler-exact-price header, and Cloudflare — acting as merchant of record — aggregates billing, charges the crawler, and pays the publisher. Publishers can allow free, charge a flat per-request rate, or block.
- discovery_endpoint
- HTTP 402 response with crawler-price header (per-request; Cloudflare-managed)
- settlement_type
- credits (Cloudflare as merchant of record; per-request billing)
- adoption_metric
- — verify-against-primary-at-build ↗ https://blog.cloudflare.com/introducing-pay-per-crawl/
- notable_adopters
{"value":"Cloudflare (operator / merchant of record)","source":"https://blog.cloudflare.com/introducing-pay-per-crawl/"}- relationships
{"predicate":"verified_by","target":"web-bot-auth","note":"Pay Per Crawl -verified_by-> Web Bot Auth (research §2 seed triple, inverse of 'Web Bot Auth verifies pay-per-crawl'): charging the right crawler depends on cryptographically verifying which agent is real."}{"predicate":"competes_with","target":"rsl","note":"Pay Per Crawl and RSL both price/gate AI access in the licensing layer; RSL is an open licensing spec, Pay Per Crawl is a Cloudflare-operated paywall."}{"predicate":"built_on","target":"x402","note":"Pay Per Crawl uses the HTTP 402 Payment Required pattern that x402 also revives."}- ideal_use_case
- A publisher behind Cloudflare who wants to monetize AI crawling per request without building payment infrastructure.
- when_to_use
- When you are on Cloudflare and want to charge AI crawlers per request (or block them) with Cloudflare handling billing and settlement.
- when_not_to_use
- When you want an open, vendor-neutral licensing declaration (use RSL) or you are not behind Cloudflare's network.
- code_example
- # Cloudflare returns to an AI crawler: HTTP/1.1 402 Payment Required crawler-price: USD 0.01 # Crawler agrees and retries: GET /article HTTP/1.1 crawler-exact-price: USD 0.01
- source
- Launch (private beta, July 1 2025), HTTP 402 + crawler-price/crawler-exact-price headers, Cloudflare as merchant of record: https://blog.cloudflare.com/introducing-pay-per-crawl/ . Seed triple (Web Bot Auth verifies pay-per-crawl): research §2.
- agent_readiness_link
- access-economics
- layer_legacy
- content