Crawler verification
A user-agent string is trivially spoofed. Confirm a crawler is genuine by its published IP ranges, reverse DNS, or a Web Bot Auth signature.
41 of 41 The AI Crawler Registry
- ClaudeBot Anthropic · training. Crawls content used to train Claude. Honors robots.txt and crawl-delay.
- Claude-User Anthropic · inference. Fetches a page in real time when a Claude user's prompt references it. User-initiated, not bulk crawling.
- Claude-SearchBot Anthropic · search. Indexes pages to power Claude's search results.
- GPTBot OpenAI · training. Crawls content that may be used to train OpenAI models.
- OAI-SearchBot OpenAI · search. Surfaces and links sites in ChatGPT search. Does not train models.
- ChatGPT-User OpenAI · inference. User-triggered fetch when a ChatGPT user or a GPT action requests a specific URL.
- PerplexityBot Perplexity · search. Indexes pages so they can be cited as sources in Perplexity answers.
- Perplexity-User Perplexity · inference. Real-time fetch in response to a user question. Per Perplexity, user-initiated fetches are not treated as automated crawling and may ignore robots.txt — verify and rate-limit at the edge if that matters to you.
- Google-Extended Google · training. A robots.txt policy token, NOT a crawler. It makes no requests and never appears in logs; disallowing it opts your content out of Gemini/Vertex training while leaving Google Search crawling untouched.
- GoogleOther Google · search. Generic Google crawler used by various teams for research and product development.
- Google-CloudVertexBot / Gemini agents Google · inference. Fetches site content on behalf of Vertex AI agents built by site owners.
- Bingbot Microsoft · search. Powers Bing and, by extension, Copilot search grounding.
- Amazonbot Amazon · search. Improves Alexa answers and supports Amazon's AI products.
- Applebot-Extended Apple · training. Policy token: disallowing it opts content out of Apple Intelligence / foundation-model training without blocking Applebot's search crawling.
- Meta-ExternalAgent Meta · training. Crawls content to train Meta's Llama models and AI products.
- CCBot Common Crawl · training. Builds the open Common Crawl corpus that many model trainers ingest downstream. Blocking CCBot blocks an upstream training-data source for the whole ecosystem.
- Bytespider ByteDance · training. Has a reputation for aggressive crawling and inconsistent robots.txt adherence. Rate-limit at the edge if it causes load.
- DuckAssistBot DuckDuckGo · inference. Fetches content for DuckDuckGo's AI assist answers.
- OAI-AdsBot OpenAI · ad-verification. Validates ad landing pages for OpenAI's advertising products. Listed alongside GPTBot/OAI-SearchBot/ChatGPT-User in OpenAI's bots documentation.
- Google-Agent Google · inference. User-triggered fetcher used by agents hosted on Google infrastructure to navigate the web and perform actions on a user's request (for example, Project Mariner / Gemini Agent). As a user-triggered fetcher, Google documents that it generally ignores robots.txt rules.
- MistralAI-User Mistral AI · inference. Fetches a page in real time when a Mistral (Le Chat) user's request references it. Per Mistral, the MistralAI-User token governs which sites these user-initiated requests can be made to.
- Diffbot Diffbot · data-aggregation. Diffbot's Crawlbot extracts and structures web content into a knowledge graph sold to customers (market intelligence, e-commerce, AI training). Registered as a 'data-provider' (Agents Welcome taxonomy extension). Diffbot documents that crawls adhere to robots.txt (disallow + crawl-delay) by default.
- Diffbot-User Diffbot · inference. Used for requests made on behalf of human users browsing URLs through Diffbot software, as distinct from Diffbot's proactive Crawlbot. Diffbot documents both 'Diffbot' and 'Diffbot-User' as robots.txt user-agents.
- ImagesiftBot ImageSift (Hive) · data-aggregation. Crawls the web for publicly available images, analyzing and indexing them to power ImageSift's web-intelligence products. Operated by ImageSift (a Hive product). Registered as a 'data-provider' (Agents Welcome taxonomy extension).
- ICC-Crawler NICT (National Institute of Information and Communications Technology) · training. Crawls data to train and support AI technologies; NICT (Japan) uses the collected data for AI and may provide it to third parties, including commercial companies. Token and operator recorded in the ai.robots.txt machine-readable registry.
- cohere-ai Cohere · inference. Retrieves data to provide responses to user-initiated prompts (Cohere products). Token and operator recorded in the ai.robots.txt machine-readable registry; the registry marks robots.txt respect as 'Unclear at this time'.
- Meta-WebIndexer Meta · search. Per Meta's documentation, the Meta-WebIndexer crawler navigates the web to improve Meta AI search result quality; allowing it in robots.txt helps Meta AI cite and link your content in its responses. Token and operator-doc reference recorded in the ai.robots.txt machine-readable registry.
- ChatGPT Atlas (agent mode) OpenAI · agentic-browsing. OpenAI's ChatGPT Atlas browser (launched 2025-10-21) embeds ChatGPT into web navigation; its 'agent mode' takes actions on the user's behalf inside the browser. As a local Chromium-based browser it presents like ordinary browser traffic with no stable AI user-agent token — included here per the agentic-browser taxonomy, verifiable by IP/signature only.
- Perplexity Comet (assistant/agent) Perplexity · agentic-browsing. Perplexity's Comet is a Chromium-based browser fork that runs locally and performs multi-tab agentic actions inside the user's session. Unlike Perplexity's cloud crawlers, it has no verifiable identity layer at the network level — included here per the agentic-browser taxonomy, verifiable by IP/signature only.
- OpenAI Operator (Computer-Using Agent) OpenAI · agentic-browsing. OpenAI's Operator (released 2025-01-23) was a browsing agent powered by the Computer-Using Agent (CUA) model that performed online tasks in a browser on the user's behalf. It was deprecated after the release of ChatGPT agent and shut down on 2025-08-31. Retained here as a deprecated agentic-browser record for history/freshness.
- Project Mariner Google · agentic-browsing. Google's Project Mariner (introduced Dec 2024 with Gemini 2.0) was an experimental web-browsing agent that navigated pages and took actions on a user's behalf via a Chrome extension. Google shut it down as a standalone product on 2026-05-04; its features moved into the Gemini API and Gemini Agent (see the Google-Agent record). Retained here as a deprecated agentic-browser record for history/freshness.
- Applebot Apple · search. Apple's crawler for Siri and Spotlight Suggestions. The separate token Applebot-Extended is used only to opt out of Apple-Intelligence training without losing search visibility.
- meta-externalfetcher Meta · inference. Fetches individual links at a user's request to support Meta AI task completion. It is user-triggered, not bulk crawling.
- meta-externalads Meta · ad-verification. Crawls the web to improve Meta's advertising and other business products.
- AI2Bot Allen Institute for AI (Ai2) · training. Collects web content to train Ai2's open language models.
- anthropic-ai Anthropic · training. Anthropic's earlier training user-agent, widely blocked in AI robots.txt files. Anthropic's current, documented training crawler is ClaudeBot — prefer targeting ClaudeBot in new rules.
- Bravebot Brave · search. Crawls to build the independent Brave Search index, which also grounds Brave's AI answers.
- kagi-fetcher Kagi · inference. Fetches pages on demand for Kagi's assistant and summarizer at a user's request; not bulk crawling.
- bedrockbot Amazon · inference. Fetches web pages for Amazon Bedrock knowledge bases and web-data connectors at a customer's request; retrieval, not bulk training.
- cohere-training-data-crawler Cohere · training. Cohere's crawler for gathering web content used to train and improve its models.
- DuckDuckBot DuckDuckGo · search. DuckDuckGo's traditional search crawler. The separate DuckAssistBot token serves its AI-assist features.