# Bytespider

> ByteDance · training

_The AI Crawler Registry · /crawlers/bytespider · [JSON](/api/crawlers/bytespider) · [all The AI Crawler Registry](/crawlers)_

- **name:** Bytespider
- **operator:** ByteDance
- **purpose:** training
- **ua_substring:** Bytespider
- **robots_token:** Bytespider
- **respects_robots:** no
- **verify:** no authoritative published range file; treat unverified Bytespider traffic with suspicion
- **notes:** Has a reputation for aggressive crawling and inconsistent robots.txt adherence. Rate-limit at the edge if it causes load.
- **canonical_name:** Bytespider
- **user_agent_token:** Bytespider
- **ua_full:** — (verify-against-primary-at-build)
- **bot_type:** training
- **bot_type_extension:** —
- **opt_out_mechanism:** robots.txt disallow (User-agent: Bytespider) — but adherence is documented as inconsistent; rate-limit / block at the edge if needed
- **published_ip_range_url:** — (verify-against-primary-at-build)
- **asn:** — (verify-against-primary-at-build)
- **reverse_dns_suffix:** — (verify-against-primary-at-build)
- **supports_web_bot_auth:** — (verify-against-primary-at-build)
- **signature_agent_domain:** — (verify-against-primary-at-build)
- **jwks_url:** — (verify-against-primary-at-build)
- **verification_methods:** user-agent-match
- **crawl_traffic_share:** 10.25% (source: https://radar.cloudflare.com/bots)
- **targeted_content_type:** HTML, text
- **documentation_url:** — (verify-against-primary-at-build)
- **first_seen_date:** — (verify-against-primary-at-build)
- **last_verified_date:** 2026-06-15
- **block_vs_allow_recommendation:** block (default) — training crawler with documented aggressive crawling and inconsistent robots.txt adherence and no authoritative IP file to verify; block/rate-limit at the edge.
- **citation_referral_value:** low (training; no direct citation or referral)
- **cloudflare_verified_category:** — (verify-against-primary-at-build)
- **status:** active
- **triples:** ["Bytespider","operated_by","ByteDance"], ["Bytespider","has_bot_type","training"], ["Bytespider","has_crawl_share","10.25% (Radar 2026-05)"]
- **attribute_sources:** {"claims":["crawl_traffic_share"],"source":"https://radar.cloudflare.com/bots","last_verified":"2026-06-15"}