Tokenization

Breaking text into tokens — the sub-word units a language model actually reads and generates — before it can be processed.

Tokens are the currency of the agentic web: clean, concise, markdown-first content tokenizes efficiently, lowering the Cost of Retrieval for every agent that reads it.

term
Tokenization
category
knowledge-memory
short_def
Breaking text into tokens — the sub-word units a language model actually reads and generates — before it can be processed.
long_def
Before a model sees text, a tokenizer splits it into tokens (whole words, word-pieces or characters) and maps each to an integer. Token counts, not character counts, drive context limits and pricing, so how efficiently your content tokenizes affects both what fits and what it costs an agent to read.
see_also
context-window token-economics embeddings
etymology_origin
— verify-against-primary-at-build ↗ https://en.wikipedia.org/wiki/Large_language_model — sub-word tokenization (BPE, WordPiece) is standard LLM practice
related_to
context-window token-economics embeddings
contrast_with
Unlike counting characters or words, tokenization counts sub-word units — the unit models bill and budget in, where one token is roughly four characters of English.
example
The word 'tokenization' may itself be several tokens; a page of prose is counted by its tokens, which is what a context window and a price-per-token bill against.
source
https://en.wikipedia.org/wiki/Large_language_model
status
active
why_it_matters
Tokens are the currency of the agentic web: clean, concise, markdown-first content tokenizes efficiently, lowering the Cost of Retrieval for every agent that reads it.
sameAs
https://en.wikipedia.org/wiki/Large_language_model
bridge_entity
token-economics
last_verified
2026-07-06
md_twin
/glossary/tokenization.md

last verified · by

← all The Agentic Web Lexicon · .md · JSON