Crawler verification
A user-agent string is trivially spoofed. Confirm a crawler is genuine by its published IP ranges, reverse DNS, or a Web Bot Auth signature.
31 of 31 The AI Crawler Registry
- ClaudeBot Anthropic · training
- Claude-User Anthropic · inference
- Claude-SearchBot Anthropic · search
- GPTBot OpenAI · training
- OAI-SearchBot OpenAI · search
- ChatGPT-User OpenAI · inference
- PerplexityBot Perplexity · search
- Perplexity-User Perplexity · inference
- Google-Extended Google · training
- GoogleOther Google · search
- Google-CloudVertexBot / Gemini agents Google · inference
- Bingbot Microsoft · search
- Amazonbot Amazon · search
- Applebot-Extended Apple · training
- Meta-ExternalAgent Meta · training
- CCBot Common Crawl · training
- Bytespider ByteDance · training
- DuckAssistBot DuckDuckGo · inference
- OAI-AdsBot OpenAI · ad-verification
- Google-Agent Google · inference
- MistralAI-User Mistral AI · inference
- Diffbot Diffbot · data-aggregation
- Diffbot-User Diffbot · inference
- ImagesiftBot ImageSift (Hive) · data-aggregation
- ICC-Crawler NICT (National Institute of Information and Communications Technology) · training
- cohere-ai Cohere · inference
- Meta-WebIndexer Meta · search
- ChatGPT Atlas (agent mode) OpenAI · agentic-browsing
- Perplexity Comet (assistant/agent) Perplexity · agentic-browsing
- OpenAI Operator (Computer-Using Agent) OpenAI · agentic-browsing
- Project Mariner Google · agentic-browsing