Fictional vocabulary
Compact profile
This prepared vocabulary contains several frequent units from the sentence.
- Models
- ␠learn
- .
Lab 02 · Prepared demonstration
One text, several tokenizations
Compare two fictional vocabularies and see how the same text becomes sequences of different lengths.
Compare two segmentations
One text, several tokenizations
Change the example and see how two prepared vocabularies represent exactly the same text with different boundaries.
Original text
Models learn.3 tokens
Fictional vocabulary
Fragmented profile
This prepared vocabulary needs more fragments to produce exactly the same text.
- Mod
- els
- ␠le
- arn
- .
A vocabulary may contain a frequent whole word or only fragments that can reconstruct it.
What this demonstration does not show
- Profiles and identifiers are fictional and prepared locally.
- No commercial or open-source tokenizer runs here.
- Real boundaries depend on the algorithm, vocabulary, version, normalization, and exact text.
- Fewer tokens do not automatically imply a better model or answer.