Adoption data · sourced from Cloudflare's public network analyses · 2024–2025
Agentic Web Adoption Report
What does the agentic web look like at the level of raw traffic? This report reads the numbers straight from Cloudflare's published network analyses. Every figure is cited, not estimated — each links its primary source and the period it covers.
The crawl-to-click gap: AI companies take far more than they send back
Cloudflare's clearest finding is the imbalance between how much an AI company crawls and how many visitors it refers. Measured as crawl requests per referral (July 2025):
| AI company | Crawls per referral (Jul 2025) | Change since Jan 2025 |
|---|---|---|
| Anthropic | 38,065 : 1 | down 86.7% (from 286,930 : 1) |
| OpenAI | 1,091 : 1 | down 10.4% (from 1,217 : 1) |
| Perplexity | 195 : 1 | up 256.7% (from 55 : 1) |
| Microsoft | 41 : 1 | up 5.7% (from 38 : 1) |
source Cloudflare, The crawl-to-click gap (data Jan–Jul 2025). The ratios are moving toward parity at the top (Anthropic's fell sharply) but remain lopsided — which is exactly the pressure behind access economics and pay-per-crawl.
Who is crawling: share shifted hard in a single year
Among AI crawlers, share of activity moved sharply from July 2024 to July 2025:
- GPTBot (OpenAI): 4.7% → 11.7%
- ClaudeBot (Anthropic): 6% → 9.9%
- Amazonbot: 10.2% → 5.9%
- Bytespider (ByteDance): 14.1% → 2.4%
source Cloudflare, The crawl-to-click gap (Jul 2024 vs Jul 2025). Each of these bots is catalogued in the AI Crawler & Agent Registry.
Why they crawl: training dominates, and it is growing
Cloudflare classifies AI crawling by purpose. Training now drives nearly 80% of AI-bot crawling, up from 72% a year earlier, while user-action and undeclared purposes together account for under 5%.
source Cloudflare, AI crawler traffic by purpose and industry (2025). This is the split that makes the training vs. search vs. user-action distinction matter for a block-or-allow decision.
It varies by industry
The mix is not uniform. In the News & Publications sector (early August 2025), OpenAI's two agents alone were dominant: GPTBot at 17.4% and the user-triggered ChatGPT-User at 14.9% of AI-bot traffic.
source Cloudflare, AI crawler traffic by purpose and industry (Aug 2025).
How to read this report
These are Cloudflare's published figures from its own network — the most complete public view of agentic-web crawling — covering 2024–2025. They are a snapshot, not a live feed: percentages shift monthly, so treat the direction (training up, crawl-to-click lopsided, GPTBot rising) as the durable signal and the exact decimals as of their stated period. Our own reference-coverage measurement lives in the State of the Agentic Web; this report reads the external network data that surrounds it.