Bias & Variance Tradeoff: Overfitting vs Underfitting
Summary of the video “Lec-43: Bias & Variance Tradeoff Explained: How to Fix Overfitting & Underfitting?” by Gate Smashers.
Bias measures training error (high bias = underfit, model too simple); variance measures test error divergence from training error (high variance = overfit, model too complex). The goal is a generalized model with low bias and low variance, achieved through techniques like cross-validation, regularization, and ensemble learning.
Foundational Concepts
What is Bias
Bias refers to errors caused by overly simplistic learning algorithms that fail to capture underlying data patterns. A model with high bias is too simple and cannot learn the training data properly, resulting in high training error.
What is Variance
Variance refers to errors caused by model sensitivity to small fluctuations in training data. High variance means the model performs well on training data but poorly on test data, indicating it has memorized rather than generalized.
Training vs Test Data Split
Data is divided into training (used to teach the model patterns) and test (unseen data used to validate generalization). Training error is measured on known data; test error reveals whether the model can predict on new data.
High Bias: The Underfit Problem
Underfit Model Defined
An underfit model fails during training itself, producing high errors on training data. The model is too simple to capture data patterns, so it will inevitably perform worse on test data.
High Bias Characteristics
High bias occurs when training error is very high. The model cannot understand its own training data, so it cannot generalize to test data. This is a sign the model is too simplistic (e.g., fitting a line to nonlinear data).
High Variance: The Overfit Problem
Overfit Model Defined
An overfit model achieves near-zero error on training data but performs poorly on test data. It has memorized training patterns including noise rather than learning generalizable patterns.
High Variance Characteristics
High variance occurs when training error is very low but test error is very high. The gap between training and test performance reveals the model is overly complex and sensitive to training data peculiarities.
Variance Example: Class Test vs GATE Exam
A student scores 98-100 on mock tests but only 30-50 on the actual GATE exam. This large gap (variance of 50-70 points) shows the student memorized specific problems rather than learning generalizable concepts.
The Generalized Model: The Goal
Low Bias and Low Variance
A generalized model achieves low error on both training and test data with minimal gap between them. It captures true patterns without memorizing noise, balancing simplicity and complexity.
Generalized Model Example: Consistent Performance
A student scores 60 on mock tests and 62 on the GATE exam. The small 2-point gap shows the model (student) learned generalizable concepts rather than memorizing specific problems.
Bias-Variance Tradeoff Summary
Models face a tradeoff: increasing complexity reduces bias but increases variance; decreasing complexity reduces variance but increases bias. The goal is finding the sweet spot where both are minimized.
Techniques to Achieve Generalization
Methods to Reduce Overfitting and Underfitting
Cross-validation, regularization (Lasso), ensemble learning (boosting and bagging), and hyperparameter tuning are techniques to balance bias and variance and achieve a generalized model.
Notable quotes
Bias refers to errors due to overly simplistic elements in the learning algorithm. — Instructor
High bias is when the model is too simple, underfit the data, it is not able to understand the pattern. — Instructor
We need a generalized model. It is neither overfit nor underfit. — Instructor
Action items
- Diagram and label the four scenarios: high bias/low variance (underfit), low bias/high variance (overfit), high bias/high variance, and low bias/low variance (generalized)
- Calculate the bias-variance gap for a model by comparing training error to test error on a dataset
- Implement cross-validation on a machine learning model to assess generalization
- Apply regularization techniques (Lasso or Ridge) to reduce overfitting on a complex model
- Experiment with ensemble methods (boosting or bagging) to balance bias and variance