Pay-per-crawl
A Cloudflare marketplace mechanism that lets a site charge AI crawlers per request — returning HTTP 402 to unpaid bots with a price in response headers.
- term
- Pay-per-crawl
- category
- commerce
- short_def
- A Cloudflare marketplace mechanism that lets a site charge AI crawlers per request — returning HTTP 402 to unpaid bots with a price in response headers.
- long_def
- Announced by Cloudflare on 1 July 2025 (private beta). A site sets a price; when an AI crawler requests a page without payment the edge answers HTTP 402 Payment Required with crawler-price headers, and Cloudflare acts as merchant of record to settle. It turns crawling from a free externality into a metered transaction.
- see_also
- —
- etymology_origin
- Announced by Cloudflare, 1 July 2025 (private beta).
- related_to
- —
- contrast_with
- Where RSL is a licensing standard (the terms), pay-per-crawl is an enforcement + settlement mechanism (the toll booth).
- example
- A news site enables pay-per-crawl so GPTBot must pay per fetch instead of crawling for free.
- source
- https://blog.cloudflare.com/introducing-pay-per-crawl/
- status
- beta
- why_it_matters
- One of the three access-economics levers (block / price / permit) and a live way publishers monetize AI crawling.
- sameAs
- —
- bridge_entity
- /access-economics
- last_verified
- 2026-06-15
- md_twin
- /glossary/pay-per-crawl.md