Start with the mechanism
Three steps to keep in mind
- 1
Normalize the text
Some tokenizers standardize spaces, case, or Unicode forms before splitting.
- 2
Find vocabulary pieces
The same word may become one token, several subwords, or byte-level pieces.
- 3
Use numerical IDs
Each token maps to an identifier understood only with that tokenizer and version.
Try it yourself
The lab isolates this mechanism with a small, transparent simulation. Everything runs locally in your browser.
Open “One text, several tokenizations”Keep these three ideas
Go deeper when you need it
Technical detailWhat the short version leaves out
Subword tokenization balances a finite vocabulary against reasonably short sequences.
Unicode, whitespace, rare words, and languages can all change how much text fits into the same token budget.
Your local progress
Status: Not started
Progress stays on this device.