# Crawler verification

> A user-agent string is trivially spoofed. Confirm a crawler is genuine by its published IP ranges, reverse DNS, or a Web Bot Auth signature.

_41 of 41 · [all The AI Crawler Registry](/crawlers)_

- [ClaudeBot](/crawlers/claudebot) — Anthropic · training. Crawls content used to train Claude. Honors robots.txt and crawl-delay.
- [Claude-User](/crawlers/claude-user) — Anthropic · inference. Fetches a page in real time when a Claude user's prompt references it. User-initiated, not bulk crawling.
- [Claude-SearchBot](/crawlers/claude-searchbot) — Anthropic · search. Indexes pages to power Claude's search results.
- [GPTBot](/crawlers/gptbot) — OpenAI · training. Crawls content that may be used to train OpenAI models.
- [OAI-SearchBot](/crawlers/oai-searchbot) — OpenAI · search. Surfaces and links sites in ChatGPT search. Does not train models.
- [ChatGPT-User](/crawlers/chatgpt-user) — OpenAI · inference. User-triggered fetch when a ChatGPT user or a GPT action requests a specific URL.
- [PerplexityBot](/crawlers/perplexitybot) — Perplexity · search. Indexes pages so they can be cited as sources in Perplexity answers.
- [Perplexity-User](/crawlers/perplexity-user) — Perplexity · inference. Real-time fetch in response to a user question. Per Perplexity, user-initiated fetches are not treated as automated crawling and may ignore robots.txt — verify and rate-limit at the edge if that matters to you.
- [Google-Extended](/crawlers/google-extended) — Google · training. A robots.txt policy token, NOT a crawler. It makes no requests and never appears in logs; disallowing it opts your content out of Gemini/Vertex training while leaving Google Search crawling untouched.
- [GoogleOther](/crawlers/googleother) — Google · search. Generic Google crawler used by various teams for research and product development.
- [Google-CloudVertexBot / Gemini agents](/crawlers/gemini-deep-research) — Google · inference. Fetches site content on behalf of Vertex AI agents built by site owners.
- [Bingbot](/crawlers/bingbot) — Microsoft · search. Powers Bing and, by extension, Copilot search grounding.
- [Amazonbot](/crawlers/amazonbot) — Amazon · search. Improves Alexa answers and supports Amazon's AI products.
- [Applebot-Extended](/crawlers/applebot-extended) — Apple · training. Policy token: disallowing it opts content out of Apple Intelligence / foundation-model training without blocking Applebot's search crawling.
- [Meta-ExternalAgent](/crawlers/meta-externalagent) — Meta · training. Crawls content to train Meta's Llama models and AI products.
- [CCBot](/crawlers/ccbot) — Common Crawl · training. Builds the open Common Crawl corpus that many model trainers ingest downstream. Blocking CCBot blocks an upstream training-data source for the whole ecosystem.
- [Bytespider](/crawlers/bytespider) — ByteDance · training. Has a reputation for aggressive crawling and inconsistent robots.txt adherence. Rate-limit at the edge if it causes load.
- [DuckAssistBot](/crawlers/duckassistbot) — DuckDuckGo · inference. Fetches content for DuckDuckGo's AI assist answers.
- [OAI-AdsBot](/crawlers/oai-adsbot) — OpenAI · ad-verification. Validates ad landing pages for OpenAI's advertising products. Listed alongside GPTBot/OAI-SearchBot/ChatGPT-User in OpenAI's bots documentation.
- [Google-Agent](/crawlers/google-agent) — Google · inference. User-triggered fetcher used by agents hosted on Google infrastructure to navigate the web and perform actions on a user's request (for example, Project Mariner / Gemini Agent). As a user-triggered fetcher, Google documents that it generally ignores robots.txt rules.
- [MistralAI-User](/crawlers/mistralai-user) — Mistral AI · inference. Fetches a page in real time when a Mistral (Le Chat) user's request references it. Per Mistral, the MistralAI-User token governs which sites these user-initiated requests can be made to.
- [Diffbot](/crawlers/diffbot) — Diffbot · data-aggregation. Diffbot's Crawlbot extracts and structures web content into a knowledge graph sold to customers (market intelligence, e-commerce, AI training). Registered as a 'data-provider' (Agents Welcome taxonomy extension). Diffbot documents that crawls adhere to robots.txt (disallow + crawl-delay) by default.
- [Diffbot-User](/crawlers/diffbot-user) — Diffbot · inference. Used for requests made on behalf of human users browsing URLs through Diffbot software, as distinct from Diffbot's proactive Crawlbot. Diffbot documents both 'Diffbot' and 'Diffbot-User' as robots.txt user-agents.
- [ImagesiftBot](/crawlers/imagesiftbot) — ImageSift (Hive) · data-aggregation. Crawls the web for publicly available images, analyzing and indexing them to power ImageSift's web-intelligence products. Operated by ImageSift (a Hive product). Registered as a 'data-provider' (Agents Welcome taxonomy extension).
- [ICC-Crawler](/crawlers/icc-crawler) — NICT (National Institute of Information and Communications Technology) · training. Crawls data to train and support AI technologies; NICT (Japan) uses the collected data for AI and may provide it to third parties, including commercial companies. Token and operator recorded in the ai.robots.txt machine-readable registry.
- [cohere-ai](/crawlers/cohere-ai) — Cohere · inference. Retrieves data to provide responses to user-initiated prompts (Cohere products). Token and operator recorded in the ai.robots.txt machine-readable registry; the registry marks robots.txt respect as 'Unclear at this time'.
- [Meta-WebIndexer](/crawlers/meta-webindexer) — Meta · search. Per Meta's documentation, the Meta-WebIndexer crawler navigates the web to improve Meta AI search result quality; allowing it in robots.txt helps Meta AI cite and link your content in its responses. Token and operator-doc reference recorded in the ai.robots.txt machine-readable registry.
- [ChatGPT Atlas (agent mode)](/crawlers/chatgpt-atlas) — OpenAI · agentic-browsing. OpenAI's ChatGPT Atlas browser (launched 2025-10-21) embeds ChatGPT into web navigation; its 'agent mode' takes actions on the user's behalf inside the browser. As a local Chromium-based browser it presents like ordinary browser traffic with no stable AI user-agent token — included here per the agentic-browser taxonomy, verifiable by IP/signature only.
- [Perplexity Comet (assistant/agent)](/crawlers/perplexity-comet) — Perplexity · agentic-browsing. Perplexity's Comet is a Chromium-based browser fork that runs locally and performs multi-tab agentic actions inside the user's session. Unlike Perplexity's cloud crawlers, it has no verifiable identity layer at the network level — included here per the agentic-browser taxonomy, verifiable by IP/signature only.
- [OpenAI Operator (Computer-Using Agent)](/crawlers/openai-operator) — OpenAI · agentic-browsing. OpenAI's Operator (released 2025-01-23) was a browsing agent powered by the Computer-Using Agent (CUA) model that performed online tasks in a browser on the user's behalf. It was deprecated after the release of ChatGPT agent and shut down on 2025-08-31. Retained here as a deprecated agentic-browser record for history/freshness.
- [Project Mariner](/crawlers/project-mariner) — Google · agentic-browsing. Google's Project Mariner (introduced Dec 2024 with Gemini 2.0) was an experimental web-browsing agent that navigated pages and took actions on a user's behalf via a Chrome extension. Google shut it down as a standalone product on 2026-05-04; its features moved into the Gemini API and Gemini Agent (see the Google-Agent record). Retained here as a deprecated agentic-browser record for history/freshness.
- [Applebot](/crawlers/applebot) — Apple · search. Apple's crawler for Siri and Spotlight Suggestions. The separate token Applebot-Extended is used only to opt out of Apple-Intelligence training without losing search visibility.
- [meta-externalfetcher](/crawlers/meta-externalfetcher) — Meta · inference. Fetches individual links at a user's request to support Meta AI task completion. It is user-triggered, not bulk crawling.
- [meta-externalads](/crawlers/meta-externalads) — Meta · ad-verification. Crawls the web to improve Meta's advertising and other business products.
- [AI2Bot](/crawlers/ai2bot) — Allen Institute for AI (Ai2) · training. Collects web content to train Ai2's open language models.
- [anthropic-ai](/crawlers/anthropic-ai) — Anthropic · training. Anthropic's earlier training user-agent, widely blocked in AI robots.txt files. Anthropic's current, documented training crawler is ClaudeBot — prefer targeting ClaudeBot in new rules.
- [Bravebot](/crawlers/bravebot) — Brave · search. Crawls to build the independent Brave Search index, which also grounds Brave's AI answers.
- [kagi-fetcher](/crawlers/kagi-fetcher) — Kagi · inference. Fetches pages on demand for Kagi's assistant and summarizer at a user's request; not bulk crawling.
- [bedrockbot](/crawlers/bedrockbot) — Amazon · inference. Fetches web pages for Amazon Bedrock knowledge bases and web-data connectors at a customer's request; retrieval, not bulk training.
- [cohere-training-data-crawler](/crawlers/cohere-training-data-crawler) — Cohere · training. Cohere's crawler for gathering web content used to train and improve its models.
- [DuckDuckBot](/crawlers/duckduckbot) — DuckDuckGo · search. DuckDuckGo's traditional search crawler. The separate DuckAssistBot token serves its AI-assist features.