Skip to content
Transformers & LLMs

From Words to Vectors

Tokenization

6 min+60 XPStep 1/10

Models don't see words — they see tokens

Before an LLM reads anything, a tokenizer chops text into tokens — subword chunks like 'un', 'believ', 'able'. Common words get one token; rare words get split.

Every model has a vocabulary of ~50–100k tokens, and a context window limiting how many tokens it can consider at once.

insight

Tokenization explains famous LLM quirks: why they struggle counting letters ('strawberry' is one token, not letters) and why API pricing is per-token.