# Search-index crawlers

> Crawlers that index pages to answer queries inside an AI search product.

_10 of 41 · [all The AI Crawler Registry](/crawlers)_

- [Claude-SearchBot](/crawlers/claude-searchbot) — Anthropic · search. Indexes pages to power Claude's search results.
- [OAI-SearchBot](/crawlers/oai-searchbot) — OpenAI · search. Surfaces and links sites in ChatGPT search. Does not train models.
- [PerplexityBot](/crawlers/perplexitybot) — Perplexity · search. Indexes pages so they can be cited as sources in Perplexity answers.
- [GoogleOther](/crawlers/googleother) — Google · search. Generic Google crawler used by various teams for research and product development.
- [Bingbot](/crawlers/bingbot) — Microsoft · search. Powers Bing and, by extension, Copilot search grounding.
- [Amazonbot](/crawlers/amazonbot) — Amazon · search. Improves Alexa answers and supports Amazon's AI products.
- [Meta-WebIndexer](/crawlers/meta-webindexer) — Meta · search. Per Meta's documentation, the Meta-WebIndexer crawler navigates the web to improve Meta AI search result quality; allowing it in robots.txt helps Meta AI cite and link your content in its responses. Token and operator-doc reference recorded in the ai.robots.txt machine-readable registry.
- [Applebot](/crawlers/applebot) — Apple · search. Apple's crawler for Siri and Spotlight Suggestions. The separate token Applebot-Extended is used only to opt out of Apple-Intelligence training without losing search visibility.
- [Bravebot](/crawlers/bravebot) — Brave · search. Crawls to build the independent Brave Search index, which also grounds Brave's AI answers.
- [DuckDuckBot](/crawlers/duckduckbot) — DuckDuckGo · search. DuckDuckGo's traditional search crawler. The separate DuckAssistBot token serves its AI-assist features.