Cloudflare AI Crawl Control reviewed: see, allow, block and charge AI crawlers

Cloudflare AI Crawl Control (formerly AI Audit) lets a site owner see which AI services fetch their content, allow or block individual crawlers with granular policies, and optionally charge for access via pay-per-crawl — all from the CDN edge in front of the site.

What does Cloudflare AI Crawl Control do?

Cloudflare AI Crawl Control lets a site owner monitor and control how AI services access the site's content, and Cloudflare lists it as available on all plans (Cloudflare, 2026). Its documentation, last updated 14 August 2026, names five capabilities: see which AI services access your content, control access with granular policies, monitor robots.txt compliance, explore monetization options, deploy with zero configuration. Enforcement runs at Cloudflare's edge, before a request reaches the origin.

Where does Cloudflare AI Crawl Control fit in the agentic web?

Cloudflare AI Crawl Control is an access-control and access-economics tool: it serves the access-control dimension of agent readiness, not content or discoverability. It operationalizes two Almanac standards: pay-per-crawl, which prices AI access at the edge, and Web Bot Auth, which verifies which crawler is real. The crawlers it lists are the entities catalogued in the AI Crawler & Agent Registry, and Cloudflare Radar's crawl figures feed the State of the Agentic Web dataset: in May 2026, GPTBot 11.48 %, Bytespider 10.25 %, Applebot 7.01 % and Claude-SearchBot 2.22 % of observed AI-crawler requests (last verified 2026-06-15).

Our take

The default place to enforce an AI-access decision if your site already sits behind Cloudflare; Radar is a canonical, citable adoption source. It tells you who is crawling and lets you gate them; it does not tell you whether your own site is agent-ready. Pair it with a readiness check, then decide allow, block or charge per crawler with the access-economics guide.

How does Cloudflare pay-per-crawl charge an AI crawler?

Pay-per-crawl answers an unpaid request with HTTP 402 Payment Required and a crawler-price header; the crawler accepts by repeating the request with crawler-exact-price, or bids ahead with crawler-max-price and is confirmed with crawler-charged (Cloudflare, July 2025). The private beta opened on 1 July 2025 with Cloudflare as merchant of record. The site owner sets one flat per-request price for the domain and chooses Allow, Charge or Block per crawler; a paying crawler must authenticate with Web Bot Auth (Ed25519 keys plus signed signature-agent, signature-input and signature headers) to prevent spoofing. The Almanac's Pay Per Crawl record (status “live (beta, 2025)”, last verified 2026-06-15) and this site's x402 demo on /api/premium/playbook, which also answers HTTP 402, let you compare both 402-based mechanisms.

Does Cloudflare AI Crawl Control tell you whether your site is agent-ready?

No. AI Crawl Control governs access; readiness is scored by a separate Cloudflare product, the Agent Readiness score, published on 17 April 2026 with a public scanner at isitagentready.com (Cloudflare, 2026). The score grades discoverability, content, bot access control and capabilities, and references AI Crawl Control only as one bot-access-control mechanism (Cloudflare, April 2026). A site can pass every AI Crawl Control policy and still fail three of the four categories.

Cloudflare AI Crawl Control — frequently asked questions

Which Cloudflare plans include AI Crawl Control?

All plans, according to Cloudflare's documentation, last updated 14 August 2026. Pay-per-crawl, the charging feature, is separate: it is in private beta with Cloudflare as merchant of record (Cloudflare, 2025).

What was Cloudflare AI Crawl Control called before?

AI Audit. Cloudflare's documentation names the product AI Crawl Control (formerly AI Audit).

Which HTTP status code does Cloudflare pay-per-crawl use?

HTTP 402 Payment Required. The response carries a crawler-price header, the crawler confirms with crawler-exact-price, and it must authenticate with Web Bot Auth before it can be charged (Cloudflare, July 2025).

Does Cloudflare AI Crawl Control measure agent readiness?

No. Readiness is scored by Cloudflare's separate Agent Readiness score, published on 17 April 2026 with a public scanner at isitagentready.com, which grades discoverability, content, bot access control and capabilities; AI Crawl Control is one bot-access-control mechanism.

Does Cloudflare AI Crawl Control work for a site that is not on Cloudflare?

No. Enforcement runs at Cloudflare's edge, before a request reaches the origin, so the site has to be on Cloudflare.

← Tooling Landscape · .md