# Pay-per-crawl

> A Cloudflare marketplace mechanism that lets a site charge AI crawlers per request — returning HTTP 402 to unpaid bots with a price in response headers.

_The Agentic Web Lexicon · /glossary/pay-per-crawl · [JSON](/api/glossary/pay-per-crawl) · [all The Agentic Web Lexicon](/glossary)_

- **term:** Pay-per-crawl
- **category:** commerce
- **short_def:** A Cloudflare marketplace mechanism that lets a site charge AI crawlers per request — returning HTTP 402 to unpaid bots with a price in response headers.
- **long_def:** Announced by Cloudflare on 1 July 2025 (private beta). A site sets a price; when an AI crawler requests a page without payment the edge answers HTTP 402 Payment Required with crawler-price headers, and Cloudflare acts as merchant of record to settle. It turns crawling from a free externality into a metered transaction.
- **see_also:** —
- **etymology_origin:** Announced by Cloudflare, 1 July 2025 (private beta).
- **related_to:** —
- **contrast_with:** Where RSL is a licensing standard (the terms), pay-per-crawl is an enforcement + settlement mechanism (the toll booth).
- **example:** A news site enables pay-per-crawl so GPTBot must pay per fetch instead of crawling for free.
- **source:** https://blog.cloudflare.com/introducing-pay-per-crawl/
- **status:** beta
- **why_it_matters:** One of the three access-economics levers (block / price / permit) and a live way publishers monetize AI crawling.
- **sameAs:** —
- **bridge_entity:** /access-economics
- **last_verified:** 2026-06-15
- **md_twin:** /glossary/pay-per-crawl.md