RL, Kernels, Quantization & Agents: Deep Dive
A comprehensive workshop covering reinforcement learning fundamentals (reward functions, PPO, GRPO), the evolution from base models to chat models via SFT and RL, practical quantization strategies for vision and language models, and hands-on demonstrations using open-source tools. Key insight: RL success depends more on well-designed reward functions than algorithmic sophistication.
The AI Evolution: From Base Models to Reasoning
Training Data Scaling Across Model Generations
Llama 1 trained on 1.4 trillion tokens; Gemma 3 on 14 trillion tokens; Llama 4 on 30 trillion tokens. Larger models and more training data correlate with lower loss and improved performance, but the field has shifted focus from pure scale to training methodology.
The Open Source Drought and Recovery
September 2024 marked an 'open source drought' when OpenAI released O1 Preview with reasoning capabilities that shocked the community. For four months, open source appeared unable to replicate this leap until DeepSeek R1 (January 2025) proved reasoning could be trained in open models, revitalizing the ecosystem.
The Training Pipeline: From Pre-training to Reasoning
Modern LLM training follows: pre-training (next-token prediction on massive data), mid-training (higher-quality data, context extension), supervised fine-tuning (instruction following), post-training (preference tuning, DPO, RLHF), and reinforcement learning with verifiable rewards (RLVR). Each stage moves the model through parameter space toward better performance.
Reinforcement Learning Fundamentals
RL vs Pre-training: Known vs Unknown Answers
Pre-training knows the correct next word in advance (maximize probability). RL does not know the best action beforehand; it only receives a reward signal at the end. RL must maximize reward by exploring and learning which actions lead to better outcomes, making it fundamentally different from supervised learning.
Agents and Environments in Language Models
In traditional RL (Pac-Man, Mario), the agent takes actions and the environment state changes continuously. For language models, the 'agent' is the LLM itself, the 'environment' is the inference space, and crucially, there is no continuous state loop—each prompt is independent. This structural difference simplifies RL application to LLMs.
Reward Function Design: The Hardest Part
RL success depends primarily on well-designed reward functions, not the algorithm. For math: distance-based scoring (e.g., |predicted - correct| / correct) works better than binary. For code: check execution, imports, syntax. For subjective tasks: use LLM-as-judge or regex matching. The algorithm is secondary; the reward function is primary.
Advantage: Reward Relative to Baseline
RL maximizes advantage (reward minus baseline/average reward), not raw reward. This prevents the model from overfitting to specific questions. If advantage > 0, the action is better than average and should be reinforced; if < 0, it should be suppressed. The baseline is typically estimated by a value model.
PPO and GRPO Algorithms
PPO: Clipping and Trust Regions
Proximal Policy Optimization (PPO) prevents large gradient updates by clipping the likelihood ratio between old and new policy probabilities within an epsilon range (e.g., 0.2). It also includes KL divergence penalty to keep the model close to the reference model, and a value model to estimate baseline rewards. These mechanisms reduce overfitting and training instability.
GRPO: Removing the Value Model
Group Relative Policy Optimization (GRPO) deletes the value model entirely and replaces it with group statistics. For each question, sample multiple completions (e.g., 4), compute mean and standard deviation of rewards, then use z-score (reward - mean) / std as the normalized advantage. This reduces parameters, memory, and compute while maintaining performance.
Inference Sampling and Temperature
GRPO requires high temperature (1.0–1.5) and top-p sampling to maximize output diversity. If temperature is 0, all samples are identical and the algorithm cannot learn. Min-p (e.g., 0.1) prevents degenerate low-probability tokens. Variability in outputs is essential for RL to discover good solutions.
Practical RL Implementation
System Prompts and Chat Templates
Base models require explicit chat templates to understand conversation format. A system prompt guides reasoning: 'Think about the problem and provide your working out between [REASONING_START] and [REASONING_END]. Provide your solution between [SOLUTION_START] and [SOLUTION_END].' These are customizable; DeepSeek's format is not mandatory.
Priming: Supervised Fine-tuning Before RL
Starting RL directly on a base model often yields zero rewards forever (bad luck). Priming with 100–7,000 supervised fine-tuning examples teaches the model basic instruction-following first. This dramatically improves the chances of RL finding positive rewards. Even 118 examples suffice; more is better but not always necessary.
Reward Function Examples
Format reward: +3 if output matches regex (e.g., contains [SOLUTION_START] and [SOLUTION_END]), else 0. Math reward: +5 if answer exact, else (predicted / correct) * 5 - 2.5 for distance-based scoring. Keyword reward: +0.5 if keyword present, -1 if absent. Combine multiple reward functions; scale them consistently (e.g., all ±1 to ±5 range).
Training Dynamics: Reward Spikes and Convergence
Early RL training shows mostly negative or zero rewards. By chance, a positive reward spike appears (e.g., +13). GRPO amplifies this. Over time, rolling average reward increases. Completion length (reasoning trace) typically grows. KL divergence increases as the model diverges from the base model. Training is inherently stochastic; bad luck is common.
Quantization: Reducing Model Size
Extreme Quantization with Minimal Accuracy Loss
DeepSeek R1 (730 GB) quantized to 1.5-bit achieves 140 GB with minimal accuracy drop. Llama 4 Scout at 1-bit shows only 1% accuracy loss vs full precision (80% vs 81% on MMLU-5-shot). The key: selectively quantize non-critical layers while preserving attention, shared experts, and early layers in higher precision (dynamic quantization).
Layer-Wise Quantization Strategy
Not all layers tolerate quantization equally. First few layers, attention layers, and shared experts must remain in higher precision. Mixture-of-experts layers can be heavily quantized. Identify critical layers by analyzing activation and weight quantization error plots—large spikes indicate layers to preserve. This varies per model architecture.
Vision Model Quantization Example
Quen 2B vision model: naive 4-bit quantization (1.36 GB) produces garbage output ('coastal area' instead of 'train'). Increasing just the first few layers to 8-bit (1.8 GB total) restores accuracy ('train traveling on tracks'). Vision modules are more sensitive to quantization than language modules.
The Superweights Paper: Critical Unquantizable Values
Research shows that certain small (not large) values in down-projection layers are critical. Zeroing even one of these values dramatically reduces accuracy. These are not outliers; they are specific, low-magnitude parameters. Quantization must preserve these superweights, not just avoid large outliers.
Future Quantization: Float 4 and Beyond
GPU speed improvements historically came from reducing numerical precision (float32 → float16 → bfloat16 → float8 → float4). Each halving roughly doubles speed. Float4 is likely the final precision frontier; further gains require architectural innovation, not precision reduction. Nvidia's Blackwell supports MXFP4 format.
Optimization and Tools
torch.compile: The Biggest Training Speedup
Wrapping PyTorch functions in torch.compile often reduces memory usage and speeds training, though not always. It has 10+ pages of tunable options. Using torch.compile effectively is one of the highest-impact optimizations available. Enable it by default on all training functions.
Memory Optimization: Weight Sharing and Gradient Checkpointing
Unsloth reduces memory by 50% by sharing VLM inference weights directly with training (no copy). Async offloaded gradient checkpointing further reduces memory. LoRA (low-rank adaptation) avoids fine-tuning all parameters. Gradient accumulation (e.g., batch_size=1, accumulation_steps=16) simulates larger batches without memory spike.
Free GPU Resources: Colab and Kaggle
Google Colab offers free GPUs (limited). Kaggle provides 30 hours of free GPU per week with no restrictions. Both support RL training on models up to 14B parameters. Unsloth notebooks demonstrate GRPO on free Colab, making advanced RL accessible without paid compute.
Open Source Ecosystem: Unsloth, VLM, TRL
Unsloth: fast fine-tuning and GRPO on consumer GPUs. VLM: efficient model serving. TRL: RL training framework. Unsloth's advantage: collocates inference and training on same GPU, sharing weights and reducing communication overhead. 10M+ monthly downloads on Hugging Face; 40k GitHub stars.
Research Directions and Open Questions
What Comes After Reasoning?
Reasoning (via RL) was a major capability jump. The question remains: what's next? DeepSeek R1 paper suggests models already possess reasoning; RL accentuates it. Whether new step-function improvements exist is unknown. The field may plateau or discover new paradigms. Open question for the community.
Model Learning vs Circuit Accentuation
Two schools of thought: (1) RL accentuates existing circuits in the model (e.g., addition already exists, RL makes it more prevalent). (2) RL teaches genuinely new capabilities. Most evidence supports (1), but the distinction remains unclear. Sparse weight updates suggest (1), but long-term dynamics may reveal (2).
Scaling Reward Functions Across Domains
Currently, verifiable rewards work well for math and code. Subjective tasks (summaries, legal reasoning, creative writing) are harder. LLM-as-judge can work but breaks down over time. Generating and verifying reward functions automatically is an active research area. Open source community could accelerate this by crowdsourcing reward functions.
Parameter Efficiency in RL
Most large labs fine-tune all parameters during RL. Research shows many updates are near-zero, suggesting parameter-efficient methods (LoRA, adapters) could work. Trade-off: efficiency vs capability. Unclear which is optimal; empirical testing required.
Notable quotes
All of AI is about efficiency. What is more efficient? That's the whole goal. — Daniel Han
The hardest part is actually the reward function itself. The algorithm is useless. Who cares about the algorithm? — Daniel Han
It's all luck. We're just praying that there's going to be some positive reward somewhere in the model. — Daniel Han
Action items
- Install Unsloth and VLM for efficient RL training on consumer GPUs.
- Design custom reward functions for your task (format, math, code execution, or LLM-as-judge).
- Use Colab or Kaggle free GPUs to experiment with GRPO on 3B–14B models; start with instruct models to skip SFT priming.
- Set temperature to 1.0–1.5 and min-p to 0.1 for inference sampling to maximize output diversity.
- Wrap training functions in torch.compile and tune its options for memory and speed gains.
- For vision models, identify critical layers via quantization error plots and preserve them in higher precision.
- Start with supervised fine-tuning (100–7,000 examples) before RL to prime the model and avoid zero-reward loops.
- Monitor rolling average reward, KL divergence, and completion length during training; expect stochastic spikes.
- Follow Nathan Lambert's RHF book and Twitter for latest research; avoid chasing every paper (many are rebuttals).
- Combine multiple reward functions and normalize their scales (e.g., all ±1 to ±5 range) for balanced training.