Search-index crawlers
Crawlers that index pages to answer queries inside an AI search product.
10 of 41 The AI Crawler Registry
- Claude-SearchBot Anthropic · search. Indexes pages to power Claude's search results.
- OAI-SearchBot OpenAI · search. Surfaces and links sites in ChatGPT search. Does not train models.
- PerplexityBot Perplexity · search. Indexes pages so they can be cited as sources in Perplexity answers.
- GoogleOther Google · search. Generic Google crawler used by various teams for research and product development.
- Bingbot Microsoft · search. Powers Bing and, by extension, Copilot search grounding.
- Amazonbot Amazon · search. Improves Alexa answers and supports Amazon's AI products.
- Meta-WebIndexer Meta · search. Per Meta's documentation, the Meta-WebIndexer crawler navigates the web to improve Meta AI search result quality; allowing it in robots.txt helps Meta AI cite and link your content in its responses. Token and operator-doc reference recorded in the ai.robots.txt machine-readable registry.
- Applebot Apple · search. Apple's crawler for Siri and Spotlight Suggestions. The separate token Applebot-Extended is used only to opt out of Apple-Intelligence training without losing search visibility.
- Bravebot Brave · search. Crawls to build the independent Brave Search index, which also grounds Brave's AI answers.
- DuckDuckBot DuckDuckGo · search. DuckDuckGo's traditional search crawler. The separate DuckAssistBot token serves its AI-assist features.