How themachine readsyour words.

Before a language model does anything, it chops your text into tokens. Different methods chop differently. Paste a sentence and watch four classic tokenizers cut the same text four different ways.