AI2Bot
Allen Institute for AI (Ai2) · training. Collects web content to train Ai2's open language models.
- name
- AI2Bot
- operator
- Allen Institute for AI (Ai2)
- purpose
- training
- ua_substring
- AI2Bot
- robots_token
- AI2Bot
- respects_robots
- yes
- verify
- Ai2 documents the crawler and states its UA string may be used to filter or reject traffic; there is no published IP file, so match the UA (see documentation_url).
- notes
- Collects web content to train Ai2's open language models.
- canonical_name
- AI2Bot
- user_agent_token
- AI2Bot
- ua_full
- Mozilla/5.0 (compatible) AI2Bot (+https://www.allenai.org/crawler) source
- bot_type
- training
- bot_type_extension
- —
- opt_out_mechanism
- robots.txt disallow (User-agent: AI2Bot); the related tokens Ai2Bot-Dolma and AI2Bot-DeepResearchEval cover other Ai2 uses
- published_ip_range_url
- — verify-against-primary-at-build ↗ https://allenai.org/crawler
- asn
- — verify-against-primary-at-build ↗ https://allenai.org/crawler
- reverse_dns_suffix
- — verify-against-primary-at-build ↗ https://allenai.org/crawler
- supports_web_bot_auth
- — verify-against-primary-at-build ↗ https://allenai.org/crawler
- signature_agent_domain
- — verify-against-primary-at-build ↗ https://allenai.org/crawler
- jwks_url
- — verify-against-primary-at-build ↗ https://allenai.org/crawler
- verification_methods
user-agent-string- crawl_traffic_share
- — verify-against-primary-at-build ↗ https://radar.cloudflare.com/bots
- targeted_content_type
- HTML, text
- documentation_url
- https://allenai.org/crawler
- first_seen_date
- — verify-against-primary-at-build ↗ https://allenai.org/crawler
- last_verified_date
- 2026-07-06
- block_vs_allow_recommendation
- conditional — trains open models; block via robots.txt to opt out.
- citation_referral_value
- low (training; does not itself cite or refer)
- cloudflare_verified_category
- — verify-against-primary-at-build ↗ https://radar.cloudflare.com/bots
- status
- active
- triples
["AI2Bot","operated_by","Allen Institute for AI (Ai2)"]["AI2Bot","has_bot_type","training"]["AI2Bot","verified_via","user-agent-string"]- attribute_sources
{"claims":["user_agent_token","robots_token"],"source":"https://allenai.org/crawler","last_verified":"2026-07-06"}{"claims":["operator","purpose","documentation_url","respects_robots"],"source":"https://allenai.org/crawler","last_verified":"2026-07-06"}
last verified · by Özden Erdinc