Build a GPT Tokenizer from Scratch
Tokenization converts raw text into integer tokens that feed into language models. This video builds byte-pair encoding (BPE) from first principles, explores GPT-2 and GPT-4 tokenizers, covers sentence piece, and reveals why tokenization causes spelling failures, poor non-English support, arithmetic errors, and bizarre edge cases like the 'solid gold Magikarp' bug.
Why Tokenization Matters
Tokenization is the hidden source of LLM weirdness
Many apparent failures of large language models—poor spelling, weak non-English support, bad arithmetic, Python inefficiency, and bizarre edge cases—actually trace back to how text is tokenized, not the model architecture itself. Tokenization deserves careful study because it is the fundamental interface between raw text and neural networks.
Tokens are the atomic unit of LLMs
Everything in a language model operates in units of tokens. The context window is measured in tokens, embeddings are indexed by token ID, and the output layer predicts the next token. Tokens are not characters; they are variable-length chunks of text that the tokenizer creates.
From Characters to Bytes to Tokens
Naive character-level tokenization is inefficient
A simple approach tokenizes each character separately, creating a vocabulary of ~65 characters for English text. This leads to very long token sequences and wastes context window. Modern tokenizers use larger chunks to compress text and allow the model to see more context.
Unicode code points are not a stable tokenization target
Python strings are sequences of Unicode code points (~150,000 possible characters across 161 scripts). While code points are stable integers, the Unicode standard is actively maintained and evolves, making it an unstable foundation for a fixed tokenizer vocabulary.
UTF-8 encoding is the standard but creates variable-length byte sequences
UTF-8 encodes each Unicode code point as 1–4 bytes. It is backward-compatible with ASCII and is the most common encoding on the internet. However, using raw UTF-8 bytes as tokens would require a vocabulary of only 256, forcing very long token sequences.
Byte-Pair Encoding Algorithm
BPE iteratively merges the most frequent byte pair
Start with a vocabulary of 256 raw bytes. Repeatedly find the most common consecutive pair of tokens in the data, create a new token for that pair, and replace all occurrences. After many iterations, you have a compressed vocabulary and shorter token sequences.
BPE training produces a merges table that defines the tokenizer
The tokenizer is fully defined by a merges dictionary mapping (child1, child2) → parent_token_id. This table records the order and identity of all merges performed during training. Given this table, you can both encode (text → tokens) and decode (tokens → text).
Compression ratio improves with more merges
Starting with 24,000 bytes of raw UTF-8 text and performing 20 merges reduces the sequence to 19,000 tokens, a compression ratio of ~1.27. More merges yield higher compression but also a larger vocabulary.
GPT-2 Tokenizer: Chunking and Special Rules
GPT-2 uses regex to chunk text before BPE to prevent unwanted merges
Instead of applying BPE directly to the entire text, GPT-2 first splits the text using a regex pattern that separates letters, numbers, punctuation, and whitespace into independent chunks. BPE is then applied within each chunk, preventing merges across category boundaries (e.g., 'e' never merges with a space to form 'e ').
GPT-2 vocabulary size is 50,257 (256 bytes + 50,000 merges + 1 special token)
GPT-2 starts with 256 raw byte tokens, performs 50,000 BPE merges, and adds one special token (end-of-text), totaling 50,257 tokens. The end-of-text token marks document boundaries in training data.
Apostrophes are handled inconsistently due to case sensitivity
The GPT-2 regex pattern matches specific apostrophe forms (e.g., U+0027) in lowercase but not uppercase, and only for certain contractions. This causes 'house' and 'House' to tokenize differently, and non-ASCII apostrophes (e.g., U+2019) are not recognized, creating inconsistent tokenization across case and Unicode variants.
GPT-4 Tokenizer: Improvements Over GPT-2
GPT-4 vocabulary size is ~100,000 (double GPT-2)
GPT-4 uses approximately 100,000 tokens instead of GPT-2's 50,257. For the same text, GPT-4 produces roughly half as many tokens, allowing the model to attend to twice as much context before predicting the next token.
GPT-4 handles whitespace in Python much more efficiently
GPT-2 tokenizes each space in Python indentation as a separate token (token 220), bloating the sequence. GPT-4 groups multiple spaces into single tokens (e.g., 4 spaces → 1 token, 7 spaces → 1 token), dramatically reducing sequence length for indented code.
GPT-4 regex pattern is case-insensitive and handles contractions consistently
GPT-4 uses the re.IGNORECASE flag, so contractions like 'House' and 'house' tokenize identically. The pattern also limits numeric tokens to 1–3 digits, preventing very long number tokens.
Tokenizer Training vs. Inference
Tokenizer training is a separate preprocessing stage from LLM training
The tokenizer is trained once on a training corpus using BPE, producing a fixed merges table and vocabulary. This is completely independent of LLM training. The tokenizer then serves as a preprocessing layer that converts all raw text to token sequences before the LLM ever sees it.
Tokenizer training set can differ from LLM training set
To improve tokenization for non-English languages and code, the tokenizer training set can include more diverse languages and programming code than the LLM training set. This ensures that rare languages and code are tokenized more densely, reducing sequence length and improving LLM performance on those domains.
Encoding and decoding are inverse operations (mostly)
Given a string, encode() produces a token sequence. Given a token sequence, decode() recovers the original string. However, not all token sequences are valid UTF-8, so decode() must handle errors gracefully (e.g., replace invalid bytes with a replacement character).
Sentence Piece: An Alternative Tokenizer
Sentence Piece performs BPE on Unicode code points, not bytes
Unlike GPT tokenizers (which merge UTF-8 bytes), Sentence Piece merges Unicode code points directly. For rare code points not in the training set, it falls back to UTF-8 byte encoding if byte_fallback=true, adding special byte tokens to the vocabulary.
Sentence Piece is used by Llama 2 and other models
Sentence Piece is a popular open-source tokenizer library that supports both training and inference. It is used by Meta's Llama 2 and other models. Unlike Tik Token (inference-only), Sentence Piece allows you to train your own tokenizer.
Sentence Piece has many configuration options and historical baggage
Sentence Piece includes numerous settings (character coverage, sentence length, normalization rules, digit splitting, etc.) that can be confusing and error-prone. The documentation is sparse, making it easy to misconfigure. The concept of 'sentences' as training units is also somewhat artificial for LLM use cases.
Sentence Piece adds a dummy space prefix to normalize word boundaries
With add_dummy_prefix=true, Sentence Piece prepends a space to the input text during encoding. This ensures that words at the beginning of text are tokenized the same way as words in the middle, improving consistency.
Special Tokens and Model Surgery
Special tokens delimit documents and conversations
Special tokens like <|endoftext|> mark document boundaries in training data, signaling to the model that what follows is unrelated to the previous document. In chat models, special tokens delimit user and assistant messages (e.g., <|im_start|>, <|im_end|>).
Adding special tokens requires extending embedding and output layers
When you add a new special token, you must resize the token embedding matrix (add a row) and the final output projection layer (add a column). These new parameters are typically initialized randomly and trained from scratch, while the rest of the model can be frozen.
Special tokens enable parameter-efficient fine-tuning
By introducing new learnable tokens (e.g., 'gist tokens' for prompt compression), you can adapt a frozen LLM to new tasks without training the entire model. Only the new token embeddings are updated during fine-tuning.
Vocabulary Size Tradeoffs
Larger vocabulary reduces sequence length but increases embedding and output layer size
A larger vocabulary compresses text into fewer tokens, allowing longer context windows. However, it increases the embedding matrix (more rows) and the output projection layer (more computation), and may lead to undertrained token embeddings if tokens are rare.
State-of-the-art models use vocabularies of 50K–100K tokens
GPT-2 uses ~50K, GPT-4 uses ~100K, and Llama 2 uses ~32K. These sizes represent a practical sweet spot between compression efficiency and computational cost.
Why LLMs Fail at Spelling and Character Tasks
Long tokens prevent character-level reasoning
When a word like 'defaultstyle' is a single token, the model cannot easily reason about individual letters. Asking 'how many L's are in defaultstyle?' requires the model to decompose a single token into characters, which it rarely sees during training.
Reversing strings is difficult because tokens are not characters
Reversing 'defaultstyle' character-by-character is hard because the model operates on tokens, not characters. If the entire word is one token, the model has no way to reverse it. However, if you first list each character separated by spaces, the model can reverse the list of tokens.
Why LLMs Struggle with Non-English Languages
Non-English text is tokenized into more tokens due to less training data
The tokenizer training set is dominated by English text. As a result, English words are merged into long tokens, while non-English words are broken into shorter, more frequent tokens. A Korean sentence that translates to English may use 3× more tokens.
Longer sequences consume more context window
Because non-English text uses more tokens, it consumes more of the finite context window. The model sees less surrounding context before running out of tokens, degrading performance on non-English tasks.
Why LLMs Are Bad at Arithmetic
Numbers are tokenized arbitrarily based on merge history
Whether a 4-digit number like '677' is one token or two depends entirely on what happened to merge during tokenizer training. The model sees '127' as one token but '677' as two, making it impossible to reason consistently about digit-by-digit arithmetic.
Llama 2 mitigates this by splitting all digits
Llama 2's tokenizer (via Sentence Piece) is configured to split all digits into individual tokens. This ensures consistent digit-level representation, improving arithmetic performance.
Why GPT-2 Is Bad at Python
Python indentation spaces are tokenized individually in GPT-2
Each space in Python indentation becomes a separate token (token 220). A function with 4-space indentation uses 4 tokens just for whitespace, bloating the sequence and wasting context window.
GPT-4 improves Python by grouping spaces into single tokens
GPT-4's tokenizer merges multiple spaces into single tokens, reducing sequence length for indented code. This was a deliberate design choice by OpenAI to improve Python coding ability.
Trailing Whitespace and Partial Tokens
Trailing spaces cause tokenization inconsistencies
In GPT tokenizers, spaces are prefixes to tokens (e.g., 'o' is token 8840, but ' o' is a different token). If you add a trailing space, you create a partial token that the model has rarely seen, causing it to behave unpredictably.
Partial tokens push the model out of distribution
When you split a token (e.g., 'defaultstyle' → 'defaultsty' + 'le'), the model encounters a token prefix it never saw during training. This causes undefined behavior: the model may emit stop tokens, hallucinate, or violate safety guidelines.
The Solid Gold Magikarp Bug
Solid Gold Magikarp is a Reddit username that became a single token
The tokenizer training set included Reddit data where the username 'solid gold Magikarp' appeared frequently. This caused the tokenizer to merge it into a single token. However, this Reddit data was not included in the LLM training set.
Untrained tokens cause undefined LLM behavior
Because 'solid gold Magikarp' never appeared in the LLM training data, its token embedding was never updated and remains randomly initialized. When the model encounters this token, it produces undefined behavior: evasion, hallucinations, insults, or safety violations.
Many other trigger tokens exhibit similar behavior
Other tokens like 'at_Roth', 'streamer_bot', and others also cause the model to malfunction. These are likely tokens that appear in the tokenizer training set but not the LLM training set, leaving their embeddings untrained.
Format Efficiency and Token Economy
YAML is more token-efficient than JSON
For the same structured data, YAML uses fewer tokens than JSON (e.g., 99 tokens vs. 116 tokens). Since users pay per token and context is finite, preferring YAML over JSON can reduce costs and preserve context.
Tokenization density varies by format and language
Different representations of the same information have different token costs. Choosing efficient formats (YAML over JSON, lowercase over mixed case, etc.) can significantly reduce token usage and cost.
Building Your Own Tokenizer
MinBPE is a clean Python implementation of BPE
The MinBPE repository provides a simple, understandable implementation of byte-pair encoding that can train and inference tokenizers. It is useful for learning and for training custom vocabularies on your own data.
Tik Token is OpenAI's official tokenizer library (inference-only)
Tik Token is a fast, efficient library for encoding and decoding with GPT-2, GPT-4, and other OpenAI tokenizers. It is implemented in Rust and handles special tokens correctly. However, it does not support training.
Sentence Piece supports both training and inference
Sentence Piece is an open-source library that can train tokenizers on custom data and perform inference. It is used by Llama 2 and other models. However, it has many configuration options and can be error-prone.
Future Directions: Tokenization-Free Models
Tokenization-free models could feed raw bytes directly to Transformers
A recent paper proposes feeding raw UTF-8 bytes directly to Transformers without tokenization. This would eliminate many tokenization-related bugs and edge cases. However, it requires architectural changes (e.g., hierarchical attention) to handle the longer sequences.
Multimodal tokenization extends the concept to images, video, and audio
Modern multimodal models tokenize images into patches, videos into frames, and audio into spectrograms, then process everything as tokens in a unified Transformer. This suggests that tokenization is a general principle for feeding any modality into neural networks.
Notable quotes
Tokenization is my least favorite part of working with large language models but unfortunately it is necessary to understand in some detail. — Andrej Karpathy
A lot of the issues that may look like issues with the neural network architecture are actually issues with the tokenization. — Andrej Karpathy
Eternal glory goes to anyone who can get rid of tokenization. — Andrej Karpathy
Action items
- Clone and study the MinBPE repository to understand BPE implementation and train your own tokenizer on custom text.
- Use the Tik Tokenizer web app (tiktoken.bip.app) to visualize how different texts tokenize under GPT-2 and GPT-4 tokenizers.
- Install and experiment with Sentence Piece to train a tokenizer similar to Llama 2, paying careful attention to configuration options like byte_fallback and add_dummy_prefix.
- Implement your own encode() and decode() functions using the BPE algorithm to understand the bidirectional relationship between text and tokens.
- Test your tokenizer on non-English text, code, and numbers to observe how tokenization density varies across domains.
- Use Tik Token library (pip install tiktoken) to encode and decode text with official OpenAI tokenizers in your own projects.
- Inspect the merges table from a trained tokenizer to understand the order and priority of token merges.
- Compare token counts for the same data in different formats (JSON vs. YAML, different languages) to understand token economy.