# AI2Bot

> Allen Institute for AI (Ai2) · training. Collects web content to train Ai2's open language models.

_The AI Crawler Registry · /crawlers/ai2bot · [JSON](/api/crawlers/ai2bot) · [all The AI Crawler Registry](/crawlers)_

- **name:** AI2Bot
- **operator:** Allen Institute for AI (Ai2)
- **purpose:** training
- **ua_substring:** AI2Bot
- **robots_token:** AI2Bot
- **respects_robots:** yes
- **verify:** Ai2 documents the crawler and states its UA string may be used to filter or reject traffic; there is no published IP file, so match the UA (see documentation_url).
- **notes:** Collects web content to train Ai2's open language models.
- **canonical_name:** AI2Bot
- **user_agent_token:** AI2Bot
- **ua_full:** Mozilla/5.0 (compatible) AI2Bot (+https://www.allenai.org/crawler) (source: https://allenai.org/crawler)
- **bot_type:** training
- **bot_type_extension:** —
- **opt_out_mechanism:** robots.txt disallow (User-agent: AI2Bot); the related tokens Ai2Bot-Dolma and AI2Bot-DeepResearchEval cover other Ai2 uses
- **published_ip_range_url:** — (verify-against-primary-at-build)
- **asn:** — (verify-against-primary-at-build)
- **reverse_dns_suffix:** — (verify-against-primary-at-build)
- **supports_web_bot_auth:** — (verify-against-primary-at-build)
- **signature_agent_domain:** — (verify-against-primary-at-build)
- **jwks_url:** — (verify-against-primary-at-build)
- **verification_methods:** user-agent-string
- **crawl_traffic_share:** — (verify-against-primary-at-build)
- **targeted_content_type:** HTML, text
- **documentation_url:** https://allenai.org/crawler
- **first_seen_date:** — (verify-against-primary-at-build)
- **last_verified_date:** 2026-07-06
- **block_vs_allow_recommendation:** conditional — trains open models; block via robots.txt to opt out.
- **citation_referral_value:** low (training; does not itself cite or refer)
- **cloudflare_verified_category:** — (verify-against-primary-at-build)
- **status:** active
- **triples:** ["AI2Bot","operated_by","Allen Institute for AI (Ai2)"], ["AI2Bot","has_bot_type","training"], ["AI2Bot","verified_via","user-agent-string"]
- **attribute_sources:** {"claims":["user_agent_token","robots_token"],"source":"https://allenai.org/crawler","last_verified":"2026-07-06"}, {"claims":["operator","purpose","documentation_url","respects_robots"],"source":"https://allenai.org/crawler","last_verified":"2026-07-06"}