How LLMs Like ChatGPT Actually Work

Internet Data Collection and Filtering

LLMs start by downloading and processing internet text via datasets like Common Crawl (2.7 billion indexed web pages as of 2024). The raw data undergoes multiple filtering stages: URL blocklists remove malware and spam sites, text extraction isolates content from HTML markup, language filtering keeps high-quality English text (65%+ threshold in Fine Web), deduplication removes duplicates, and PII removal strips personal information. The result: 44 terabytes of high-quality, diverse text despite the internet's vastness.

Data processing pipeline for pre-training datasets

Tokenization: Converting Text to Numbers

Raw text must be converted into tokens—discrete symbols the neural network understands. Starting with UTF-8 encoding (0s and 1s), the Byte Pair Encoding algorithm groups common consecutive bytes into new tokens, iteratively reducing sequence length while expanding vocabulary. GPT-4 uses 100,277 tokens; this compression is crucial because sequence length is a precious computational resource. The same text can tokenize differently based on spacing and capitalization.

Vocabulary size trade-off: fewer symbols = longer sequences, more symbols = shorter sequences

Neural Network Training on Token Sequences

The pre-training stage trains a Transformer neural network to predict the next token in a sequence. Given a context window of tokens (e.g., 8,000 tokens), the network outputs probabilities for all 100,277 possible next tokens. Training iteratively adjusts billions of parameters so correct next tokens receive higher probability. The network learns statistical patterns of how tokens follow each other. GPT-2 (2019) had 1.6 billion parameters trained on 100 billion tokens; modern models have hundreds of billions of parameters trained on trillions of tokens.

Scale growth from GPT-2 to modern models

Inference: Generating Text Token-by-Token

During inference, the model generates new text by sampling from probability distributions. Given a prefix (e.g., 'hello'), it predicts probabilities for the next token, samples one according to those probabilities, appends it, and repeats. Because sampling is stochastic, the same input produces different outputs. The model never leaves the training distribution exactly—it remixes patterns statistically similar to training data but not verbatim copies.

Autoregressive text generation process

Computational Cost and GPU Infrastructure

Pre-training requires massive compute: GPT-2 cost ~$40,000 in 2019; reproducing it today costs ~$600 in 1 day due to better hardware and software. Training runs on thousands of GPUs (H100s cost $3/hour on-demand) stacked into data centers. Companies like OpenAI, Meta, and Google operate massive GPU clusters because training is the computational bottleneck. This GPU scarcity has driven Nvidia's market value to $3.4 trillion and fueled the 'gold rush' for AI infrastructure.

Training cost reduction and current GPU pricing

Base Models as Internet Simulators

Pre-training produces a 'base model'—a token simulator that generates text statistically similar to the internet. It is not an assistant; it merely continues token sequences. Base models can be prompted cleverly (e.g., few-shot examples, conversation templates) to elicit knowledge, but they lack instruction-following behavior. They memorize high-quality sources (e.g., Wikipedia) if seen multiple times during training, leading to regurgitation. Base models are rarely released because they require post-training to become useful assistants.

Human-Curated Conversation Datasets

Post-training transforms base models into assistants by fine-tuning on millions of human-written conversations. Human labelers (hired by companies like OpenAI via platforms like Upwork or Scale AI) create prompts and write ideal assistant responses following detailed labeling instructions (typically hundreds of pages). These instructions specify behavior: be helpful, truthful, and harmless. Modern datasets also use LLMs to generate synthetic conversations, which are then edited by humans. The instruct-GPT paper (2022) pioneered this approach; today's datasets like UltraChat contain millions of mostly-synthetic conversations.

Supervised fine-tuning data creation pipeline

Conversation Tokenization and Protocol

Conversations are encoded into token sequences using special tokens (e.g., GPT-4 uses 'im_start', 'im_end', 'user', 'assistant' markers) interspersed with text. A two-turn conversation becomes a flat 1D token sequence. Different LLM providers use slightly different protocols, but all convert structured conversations into tokens so the same training algorithm applies. Special tokens are new, never seen during pre-training, and help the model learn turn structure.

What You're Actually Talking To

When you interact with ChatGPT, you're not talking to a magical AI—you're talking to a statistical simulation of a human labeler who followed the company's labeling instructions. The model imitates the style, tone, and knowledge of these labelers. If your exact question appears in the training set, you may get a near-identical response. If not, the model generalizes statistically based on similar conversations. The model's cognition differs from human cognition: what's easy for humans may be hard for the model and vice versa.

Hallucinations and Mitigation

Hallucinations occur because models are trained to confidently answer questions (mimicking labelers who research answers). When asked about unknown facts, the model doesn't say 'I don't know'—it makes up plausible-sounding answers. Mitigation: empirically probe the model to find its knowledge boundaries, then add training examples where the correct answer is 'I don't know' for things outside its knowledge. Meta's Llama 3 paper describes this procedure: generate questions from random documents, test if the model knows the answers, add refusal examples for unknowns. This reduces but doesn't eliminate hallucinations.

Tool Use and Web Search

Modern assistants can use tools like web search and code interpreters to improve performance. When unsure about current facts, the model can trigger a web search. For math problems, it can write and execute code to verify answers. Tool use is itself learned during fine-tuning: the model learns to output special tokens indicating tool calls, and the system executes those calls and feeds results back into the context. This extends the model's capabilities beyond its training data.

Swiss Cheese Capability Model

LLMs have unpredictable failure modes—they excel at many tasks but randomly fail at seemingly simple ones. Example: comparing 9.11 vs. 9.9 numerically confuses the model (Bible verse associations interfere), yet it solves Olympiad problems. Reason: finite compute per token prevents large logical leaps; some token sequences require too much computation. Users should think of capabilities as Swiss cheese: lots of solid areas but arbitrary holes. Always verify outputs; don't trust blindly.

The Three Stages of Learning Analogy

Training LLMs mirrors teaching children: (1) Pre-training = reading textbooks' exposition to build knowledge; (2) Supervised fine-tuning = studying worked solutions from experts to learn imitation; (3) Reinforcement learning = solving practice problems to discover personal strategies. The first two stages are standard across all LLM providers. RL is newer, less standardized, and where reasoning emerges. Unlike imitation, RL allows models to discover solution paths humans wouldn't write.

Trial-and-Error Learning in Verifiable Domains

In RL, the model generates many candidate solutions for a problem (e.g., 1,000 math solutions per prompt), checks which reach the correct answer, and trains on successful solutions. The model practices discovering token sequences that reliably solve problems. Unlike SFT (which imitates expert solutions), RL finds solutions optimized for the model's cognition, not human cognition. It can discover approaches humans wouldn't use. This only works in verifiable domains (math, code) where correctness is checkable.

Reinforcement learning workflow for math/code problems

Emergent Reasoning and Chain-of-Thought

Deep Seek R1 paper showed that RL discovers internal reasoning strategies without explicit instruction. Models learn to re-evaluate steps, try different perspectives, backtrack, and reframe problems—behaviors resembling human internal monologue. These chains-of-thought emerge purely from optimizing for correct answers; no human hardcoded them. Response length increases because the model uses more tokens for thinking. This is fundamentally new: RL discovers cognitive strategies, not just imitation.

Emergent

Chain-of-thought reasoning discovered by RL without human instruction

Models learn to think step-by-step through reinforcement learning

Comparison to AlphaGo: Move 37

AlphaGo (trained with RL on Go) discovered 'Move 37'—a play humans rated 1-in-10,000 probability but was brilliant in retrospect. RL isn't constrained by human expertise; it can discover novel strategies. Similarly, reasoning models may discover analogies, thinking patterns, or even alternative languages for reasoning that humans wouldn't invent. The frontier is unknown: RL on open-domain reasoning could yield discoveries beyond human cognition.

Thinking Models: O1, O3, Deep Seek R1

Modern 'thinking' or 'reasoning' models (OpenAI's O1/O3, Deep Seek R1, Google's Gemini 2.0 Flash Thinking) are trained with RL on math and code problems. They show visible reasoning traces (or summaries thereof). These models outperform SFT-only models on hard reasoning tasks. However, RL is still experimental (as of early 2025); details are proprietary and not fully standardized. Thinking models are slower (they 'think' for seconds to minutes) but more accurate on complex problems. They're best for math, code, and reasoning; overkill for factual questions.

Current reasoning model landscape

Reinforcement Learning from Human Feedback (RLHF)

RLHF enables RL in unverifiable domains (creative writing, summarization) by training a 'reward model' to simulate human preferences. Humans rank candidate outputs (e.g., order 5 jokes best-to-worst), then a neural network learns to predict human rankings. RL then optimizes against this reward model instead of humans. Advantage: scales to millions of examples without human overhead. Disadvantage: reward models are gameable—RL discovers adversarial inputs that fool the model into high scores despite nonsensical outputs. RLHF is not true RL; it's limited fine-tuning that must be capped after a few hundred updates to prevent reward hacking.

RLHF pipeline for unverifiable domains

Why RLHF Isn't True RL

True RL (as in AlphaGo) has a perfect, non-gameable scoring function (win/loss). RLHF uses a neural network reward model that can be tricked. RL discovers adversarial examples—nonsensical inputs that score high—because the reward model is a massive neural network with billions of parameters. Running RL too long causes reward hacking: outputs become incoherent but score highly. Solution: cap RLHF at a few hundred updates, then stop. This is why RLHF is better thought of as fine-tuning, not true RL. True RL (on verifiable problems) can run indefinitely and discover increasingly sophisticated strategies.

Where to Find and Use Models

Proprietary models: ChatGPT (chat.openai.com), Gemini (gemini.google.com), Claude (claude.ai). Open-weights models: Together.ai and Hyperbolic host inference endpoints for Deep Seek, Llama, etc. Local inference: LM Studio lets you run smaller models on your laptop (Deep Seek distilled versions, Llama 3.2 1B). Leaderboards: LMArena ranks models by human preference; AI News newsletter covers developments; X/Twitter has active AI community. Choose based on task: GPT-4 for general use, thinking models for hard reasoning, open models for privacy/cost.

Multimodal and Long-Context Future

Future models will natively handle audio, images, and video alongside text by tokenizing them (spectrograms for audio, patches for images). This requires no algorithmic change—just more token types. Long-context is harder: current models handle 4K-200K tokens; future tasks need millions. Test-time training (updating parameters at inference, like human sleep-learning) is unexplored but necessary for long-running tasks. Agents will perform multi-step tasks over hours with human supervision, similar to human-robot ratios in factories.

Best Practices for Using LLMs

Treat LLMs as tools, not oracles. Use them for inspiration, first drafts, and brainstorming. Always verify outputs—they hallucinate, miscalculate, and fail randomly. For reasoning-heavy tasks, use thinking models (O3, Deep Seek R1). For factual questions, GPT-4 or Sonnet suffice. Check their work, own the final product. They're incredibly useful (80-90% of daily tasks) but imperfect. Combine with web search for current facts, code execution for math, and human judgment for critical decisions.