What Is Tokenization?

Before a language model can process text, it must be broken into tokens using a tokenizer. Modern tokenizers use subword algorithms like Byte Pair Encoding, which balances vocabulary size with the ability to handle any text. Common words become single tokens, while rare words are split into smaller pieces. For example, 'unhappiness' might become 'un', 'happiness' or 'un', 'happ', 'iness.' Each token is mapped to a numerical ID that the model processes. The tokenizer's vocabulary and algorithm significantly impact model performance — different models use different tokenizers, which is why the same text may have different token counts across models. Tokenization also handles special tokens for conversation structure, like end-of-sequence markers.