Building GPT from Scratch

ChatGPT is a probabilistic language model

ChatGPT generates text by predicting the next word or token based on previous context. It is probabilistic, meaning the same prompt can produce different outputs because it samples from a probability distribution rather than deterministically selecting the highest-probability token.

GPT stands for Generatively Pre-trained Transformer

The 'Transformer' architecture, introduced in the 2017 paper 'Attention is All You Need,' is the neural network at the core of GPT models. This architecture has become the foundation for most modern AI systems and was originally designed for machine translation but has since been adapted across the field.

Language models work by completing sequences

A language model takes a sequence of characters or tokens as input and predicts what comes next. It learns patterns from training data (like Shakespeare) and can generate new text by repeatedly predicting the next token and appending it to the sequence.

Start with a bigram model (simplest baseline)

A bigram model predicts the next character based only on the current character's identity, ignoring all context. While extremely simple, it provides a baseline and demonstrates the core concept of predicting sequences. Training on Shakespeare achieves a loss of approximately 2.5.

2.5

Bigram model validation loss

Baseline performance with no context

Tokenization: convert text to integers

Before training, raw text must be converted to integers via a tokenizer. Character-level tokenization maps each of the 65 unique characters in Shakespeare to an integer (0-64). More sophisticated tokenizers like GPT's use subword tokens (50,000 vocabulary), balancing sequence length and vocabulary size.

Vocabulary size trade-offs: smaller vocab means longer sequences

Batching and block size for efficient training

Instead of feeding entire texts into the model, we sample random chunks (block_size) and process multiple chunks in parallel (batch_size). A block of 9 characters contains 8 training examples because each position predicts the next character. This enables GPU parallelization and trains the model on variable context lengths (1 to block_size).

Data batching extracts multiple examples from each chunk

Token and positional embeddings

Raw token indices are converted to dense vectors via an embedding table (vocab_size × embedding_dim). Separately, positional embeddings encode the position of each token (0 to block_size). These are added together so the model knows both what token it is and where it appears in the sequence.

Self-attention enables tokens to communicate

In a bigram model, tokens do not interact. Self-attention allows each token to look at all previous tokens and gather relevant information. Each token produces a query (what am I looking for), keys (what do I contain), and values (what information do I share). Queries dot-product with keys to compute attention weights, which weight the aggregation of values.

Scaled dot-product attention prevents softmax saturation

Raw attention scores are divided by sqrt(head_size) before softmax. Without scaling, attention scores can become very large or very small, causing softmax to collapse to one-hot vectors and preventing gradient flow. Scaling preserves variance and keeps attention weights diffuse during initialization.

1 / √head_size

Scaling factor for attention scores

Prevents extreme values that would sharpen softmax too early

Causal masking prevents future tokens from leaking

In language modeling, a token at position t should not see tokens at positions t+1, t+2, etc. (the future). Causal masking sets future attention weights to negative infinity before softmax, forcing them to zero. This is called a decoder block and is essential for autoregressive generation.

Multi-head attention runs multiple attention heads in parallel

Instead of one attention mechanism, multiple independent heads run in parallel, each with a smaller head_size. For example, with embedding_dim=32 and 4 heads, each head is 8-dimensional. Outputs are concatenated back to 32 dimensions. This allows the model to attend to different types of relationships simultaneously.

Multi-head attention splits computation across parallel channels

Validation loss improves with attention

Adding a single self-attention head reduces validation loss from 2.5 (bigram) to 2.4. Adding multi-head attention further reduces it to 2.28. This demonstrates that enabling token communication substantially improves the model's ability to predict sequences.

Validation loss decreases as model complexity increases

Transformer blocks interleave communication and computation

A Transformer block contains two sub-layers: (1) multi-head self-attention for communication between tokens, and (2) a feed-forward network (MLP) for per-token computation. The feed-forward typically has an inner layer 4× the embedding dimension, allowing the model to think on information gathered via attention.

Each block alternates between communication and thinking

Residual connections enable deep networks

Residual (skip) connections add the input directly to the output of a transformation: output = input + f(input). This creates a gradient highway during backpropagation, allowing gradients to flow directly from the loss to early layers without being blocked. Residual blocks are initialized to contribute minimally at first, then gradually activate during training.

Layer normalization stabilizes training

Layer norm normalizes each token's features to zero mean and unit variance (per token, not per batch). Applied before transformations (pre-norm), it stabilizes activation magnitudes and enables training of deeper networks. Unlike batch norm, layer norm has no dependence on batch size and behaves identically at train and test time.

Dropout regularizes to prevent overfitting

Dropout randomly zeros out 20% of intermediate activations during training, effectively training an ensemble of sub-networks. At test time, all units are active. This regularization technique helps prevent overfitting, especially important when scaling up model size.

Scaling up the model dramatically improves performance

By increasing batch_size to 64, block_size to 256, embedding_dim to 384, num_heads to 6, and num_layers to 6, validation loss drops from 2.07 to 1.48—a substantial improvement. The model trains in ~15 minutes on an A100 GPU and generates recognizable Shakespeare-like text with proper formatting and character names.

Scaling hyperparameters yields 29% loss reduction

Decoder-only Transformers for language generation

A decoder-only Transformer (like GPT) uses causal masking to prevent future tokens from being seen, making it suitable for autoregressive text generation. It has no encoder and no cross-attention. This is what we built: a model that generates text unconditionally by completing sequences.

Encoder-decoder Transformers for conditional generation

The original 'Attention is All You Need' paper describes an encoder-decoder architecture for machine translation. The encoder (no causal mask) processes the source language fully. The decoder uses cross-attention to condition generation on the encoder's output. This allows conditioning on external context (e.g., translating French to English).

Pre-training: scale up on massive internet text

ChatGPT begins with pre-training a decoder-only Transformer on ~300 billion tokens from the internet. GPT-3 uses 175 billion parameters (vs. our 10 million) and trains for weeks on thousands of GPUs. The result is a document completer that can generate coherent text but does not follow instructions or answer questions helpfully.

Fine-tuning stage 1: supervised instruction following

After pre-training, the model is fine-tuned on curated examples of question-answer pairs (thousands, not billions). This teaches the model to expect a question and generate an answer, aligning it toward assistant-like behavior. Large models are sample-efficient during fine-tuning.

Fine-tuning stage 2: reward modeling and RLHF

Human raters rank multiple model responses by preference. A separate reward model is trained to predict which responses are better. Then policy gradient reinforcement learning (RLHF) optimizes the model to generate high-reward responses. This multi-step process aligns the model with human values.

RLHF aligns model behavior with human preferences

ChatGPT requires infrastructure at massive scale

Training GPT-3 requires thousands of GPUs coordinating across multiple nodes. Our 10-million-parameter model trained in ~15 minutes on a single A100 GPU. GPT-3 training takes weeks. This infrastructure complexity is a major barrier to reproducing state-of-the-art models.

NanoGPT: a minimal, clean implementation

The nanoGPT repository (available on GitHub) contains ~600 lines of code across two files: model.py (defines the Transformer) and train.py (training loop). It reproduces GPT-2's performance on the OpenWebText dataset and can be adapted to any text file, making it an educational reference implementation.

Tiny Shakespeare dataset enables fast iteration

The Tiny Shakespeare dataset is 1 MB of concatenated Shakespeare works (~1 million characters). It is small enough to train models in minutes on a single GPU, making it ideal for learning and experimentation. The same code scales to larger datasets like OpenWebText (15 GB).

1 MB

Tiny Shakespeare dataset size

Small enough for fast iteration, large enough to learn patterns

Training loop: forward pass, loss, backward pass, optimization

Each training iteration samples a batch, computes logits and cross-entropy loss, backpropagates gradients, and updates parameters using an optimizer (Adam). Validation loss is measured periodically on held-out data to detect overfitting. Learning rate decay and checkpoint saving are common practices.