Meta-WebIndexer
Meta · search
- name
- Meta-WebIndexer
- operator
- Meta
- purpose
- search
- ua_substring
- Meta-WebIndexer
- robots_token
- Meta-WebIndexer
- respects_robots
- no
- verify
- Meta publishes crawler IP ranges; confirm against those. Meta documents that allowing Meta-WebIndexer in robots.txt lets Meta AI cite and link your content.
- notes
- Per Meta's documentation, the Meta-WebIndexer crawler navigates the web to improve Meta AI search result quality; allowing it in robots.txt helps Meta AI cite and link your content in its responses. Token and operator-doc reference recorded in the ai.robots.txt machine-readable registry.
- canonical_name
- Meta-WebIndexer
- user_agent_token
- Meta-WebIndexer
- ua_full
- — verify-against-primary-at-build ↗ https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/
- bot_type
- search-index
- bot_type_extension
- —
- opt_out_mechanism
- robots.txt disallow (User-agent: Meta-WebIndexer)
- published_ip_range_url
- — verify-against-primary-at-build ↗ https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/
- asn
- — verify-against-primary-at-build ↗ https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/
- reverse_dns_suffix
- — verify-against-primary-at-build ↗ https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/
- supports_web_bot_auth
- — verify-against-primary-at-build ↗ https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/
- signature_agent_domain
- — verify-against-primary-at-build ↗ https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/
- jwks_url
- — verify-against-primary-at-build ↗ https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/
- verification_methods
published-IP-range- crawl_traffic_share
- — verify-against-primary-at-build ↗ https://radar.cloudflare.com/bots
- targeted_content_type
- HTML, text
- documentation_url
- https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/
- first_seen_date
- — verify-against-primary-at-build ↗ https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/
- last_verified_date
- 2026-06-15
- block_vs_allow_recommendation
- allow — Meta documents that allowing Meta-WebIndexer lets Meta AI cite and link your content in its answers; blocking forfeits that citation/referral.
- citation_referral_value
- high (powers Meta AI search answers; Meta documents that it cites and links allowed content)
- cloudflare_verified_category
- — verify-against-primary-at-build ↗ https://radar.cloudflare.com/bots/directory/meta-webindexer
- status
- active
- triples
["Meta-WebIndexer","operated_by","Meta"]["Meta-WebIndexer","has_bot_type","search-index"]["Meta-WebIndexer","verified_via","published-IP-range"]- attribute_sources
{"claims":["user_agent_token","robots_token","operator","documentation_url","block_vs_allow_recommendation"],"source":"https://github.com/ai-robots-txt/ai.robots.txt/blob/main/robots.json","last_verified":"2026-06-15"}