# Tokenization

> Breaking text into tokens — the sub-word units a language model actually reads and generates — before it can be processed.

_The Agentic Web Lexicon · /glossary/tokenization · [JSON](/api/glossary/tokenization) · [all The Agentic Web Lexicon](/glossary)_

- **term:** Tokenization
- **category:** knowledge-memory
- **short_def:** Breaking text into tokens — the sub-word units a language model actually reads and generates — before it can be processed.
- **long_def:** Before a model sees text, a tokenizer splits it into tokens (whole words, word-pieces or characters) and maps each to an integer. Token counts, not character counts, drive context limits and pricing, so how efficiently your content tokenizes affects both what fits and what it costs an agent to read.
- **see_also:** context-window, token-economics, embeddings
- **etymology_origin:** — (verify-against-primary-at-build)
- **related_to:** context-window, token-economics, embeddings
- **contrast_with:** Unlike counting characters or words, tokenization counts sub-word units — the unit models bill and budget in, where one token is roughly four characters of English.
- **example:** The word 'tokenization' may itself be several tokens; a page of prose is counted by its tokens, which is what a context window and a price-per-token bill against.
- **source:** https://en.wikipedia.org/wiki/Large_language_model
- **status:** active
- **why_it_matters:** Tokens are the currency of the agentic web: clean, concise, markdown-first content tokenizes efficiently, lowering the Cost of Retrieval for every agent that reads it.
- **sameAs:** https://en.wikipedia.org/wiki/Large_language_model
- **bridge_entity:** token-economics
- **last_verified:** 2026-07-06
- **md_twin:** /glossary/tokenization.md