Adoption data · sourced from Cloudflare's public network analyses · 2024–2025
Agentic Web Adoption Report
What does the agentic web look like at the level of raw traffic? This report reads the numbers straight from Cloudflare's published network analyses. Every figure is cited, not estimated — each links its primary source and the period it covers.
The crawl-to-click gap: AI companies take far more than they send back
Cloudflare's clearest finding is the imbalance between how much an AI company crawls and how many visitors it refers. Measured as crawl requests per referral (July 2025):
| AI company | Crawls per referral (Jul 2025) | Change since Jan 2025 |
|---|---|---|
| Anthropic | 38,065 : 1 | down 86.7% (from 286,930 : 1) |
| OpenAI | 1,091 : 1 | down 10.4% (from 1,217 : 1) |
| Perplexity | 195 : 1 | up 256.7% (from 55 : 1) |
| Microsoft | 41 : 1 | up 5.7% (from 38 : 1) |
source Cloudflare, The crawl-to-click gap (data Jan–Jul 2025). The ratios are moving toward parity at the top (Anthropic's fell sharply) but remain lopsided — which is exactly the pressure behind access economics and pay-per-crawl.
Who is crawling: share shifted hard in a single year
As a share of all bot traffic, the individual AI crawlers moved sharply from July 2024 to July 2025:
- GPTBot (OpenAI): 4.7% → 11.7%
- ClaudeBot (Anthropic): 6% → 9.9%
- Amazonbot: 10.2% → 5.9%
- Bytespider (ByteDance): 14.1% → 2.4%
source Cloudflare, The crawl-to-click gap (Jul 2024 vs Jul 2025). Each of these bots is catalogued in the AI Crawler & Agent Registry.
Why they crawl: training dominates, and it is growing
Cloudflare classifies AI crawling by purpose. Training now drives nearly 80% of AI-bot crawling, while user-action and undeclared purposes together account for under 5%. source Cloudflare, AI crawler traffic by purpose and industry (2025).
And that share is growing: training was 72% of AI crawling in July 2024, rising to ~79% by July 2025. source Cloudflare, The crawl-to-click gap. This training-vs-search-vs-user-action split is what makes the crawler-purpose distinction matter for a block-or-allow decision.
It varies by industry
The mix is not uniform. In the News & Publications sector (early August 2025), OpenAI's two agents alone were dominant: GPTBot at 17.4% and the user-triggered ChatGPT-User at 14.9% of AI-bot traffic.
source Cloudflare, AI crawler traffic by purpose and industry (Aug 2025).
How to read this report
These are Cloudflare's published figures from its own network — the most complete public view of agentic-web crawling — covering 2024–2025. They are a snapshot, not a live feed: percentages shift monthly, so treat the direction (training up, crawl-to-click lopsided, GPTBot rising) as the durable signal and the exact decimals as of their stated period. Our own reference-coverage measurement lives in the State of the Agentic Web; this report reads the external network data that surrounds it.