Understanding Tokenizers: The Building Blocks of AI
An interactive guide to how LLMs read text, featuring Character, Word, and Sub-word (BPE) tokenization.
When we talk to AI models like ChatGPT, we type words. But Large Language Models (LLMs) don’t actually read words the way humans do; they read tokens.
A tokenizer is the critical piece of software that acts as a translator between human text and the numbers an AI model understands. Think of it as a machine that chops a block of text into smaller, digestible pieces before feeding it into the neural network.
There are three main ways to tokenize text. Let’s explore them!
The simplest approach is to chop the text into individual characters. Every single letter, space, and punctuation mark becomes a unique token.
- Pros: The vocabulary size is extremely small (just the alphabet and symbols), meaning no “out of vocabulary” errors.
- Cons: The AI model has to work incredibly hard to understand the meaning of words, because it only sees letters. It makes the input sequences very long.
This approach chops text wherever there is a space or punctuation mark.
- Pros: It aligns well with human language. Each token carries real semantic meaning (a whole word).
- Cons: The vocabulary size becomes massive. Think about how many words exist in the English language, plus variations (run, running, ran). Also, it struggles with languages that don’t use spaces (like Japanese or Chinese).
This is the “Goldilocks” approach and what modern LLMs actually use. Algorithms like Byte-Pair Encoding (BPE) start with characters and iteratively merge the most frequently occurring pairs into sub-words.
- Common words like “the” or “token” remain as a single token.
- Rare words or names might be chopped into two or three pieces (e.g., “tokenization” might become “token” and “ization”).
- Pros: Balances vocabulary size with meaning. It can handle any weird string by falling back to smaller chunks or characters.
Don’t just take my word for it. Try it yourself!
Type any sentence in the box below to see exactly how it gets chopped up by the three different methods in real-time. Notice how the BPE tokenizer handles common words vs. complex ones!
Tokenizer Playground
Character Tokenization
0 tokensEvery single letter, space, and punctuation mark is a separate token.
Word Tokenization
0 tokensText is split by spaces and punctuation boundaries.
Sub-word Tokenization (BPE)
Used by LLMs like GPT. Common words are whole tokens, but rare words are split into chunks.
As you can see, tokenization drastically changes how much “information” is packed into a single token. When an AI company says their model supports a “128,000 token context window”, they mean 128,000 of those sub-word pieces, which usually equates to about 90,000 English words.
Understanding tokens is the first step to mastering how you interact with AI!