The Perceptron Trick: Training Weights & Bias
Summary of the video “Perceptron Trick | How to train a Perceptron | Perceptron Part 2 | Deep Learning Full Course” by CampusX.
The perceptron trick is an iterative algorithm that trains a linear classifier by randomly sampling misclassified points and updating the decision boundary (weights and bias) using a learning rate. The key insight is that when a point is on the wrong side of the line, you shift the line toward it by adding or subtracting the learning rate times the point's coordinates from the current weights.
Linear Separability & The Goal
Linearly Separable Data
Data is linearly separable when you can classify it into two classes by drawing a single line (or hyperplane in higher dimensions). The perceptron trick finds this separating line by iteratively adjusting weights and bias.
Training Goal
The perceptron training goal is to find the correct weights (w1, w2) and bias (c) such that the decision boundary correctly classifies all training points. This is done through repeated random sampling and weight updates.
Understanding Decision Regions
Positive and Negative Regions
For a line equation like 2x + 3y + 5 = 0, the positive region is where 2x + 3y + 5 > 0 (shown in blue), and the negative region is where 2x + 3y + 5 < 0. Any point can be tested against the line equation to determine which region it belongs to.
Misclassification Detection
A point is misclassified when it is on the wrong side of the decision line. For example, if a point should be negative (blue) but the model predicts it as positive (in the positive region), it is misclassified and needs correction.
Line Transformations
Three Types of Line Transformations
A line can be transformed in three ways: (1) changing the intercept c shifts the line up or down parallel to itself, (2) changing w1 rotates the line around the y-axis, and (3) changing w2 rotates the line around the x-axis. Combinations of these create the desired movement toward misclassified points.
Direction of Update
When a negative point is in the positive region, add the learning rate times the point's coordinates to the weights. When a positive point is in the negative region, subtract the learning rate times the point's coordinates. This pulls the line toward the misclassified point.
The Perceptron Trick Algorithm
Learning Rate
The learning rate is a small multiplier (typically 0.01 or similar) that controls the step size of weight updates. Rather than making large jumps that overshoot, small learning rates ensure gradual, stable convergence toward the correct decision boundary.
Core Update Rule
For each misclassified point, update weights as: w_new = w_old + learning_rate * y * x, where y is the true label (1 or -1) and x is the point's feature vector. This single formula handles both positive and negative misclassifications automatically.
Simplified Algorithm (No Explicit Condition Check)
Instead of checking if a point is misclassified before updating, simply apply the update rule to every randomly selected point. Correctly classified points will have y * (w·x + c) > 0, so the update will have minimal effect. Only misclassified points (where y * (w·x + c) < 0) will cause significant weight changes.
Code Implementation
Data Structure Setup
Organize training data as a matrix where each row is a student record: the first two columns are features (x1, x2), and the last column is the binary label (1 for placed, 0 for not placed). Initialize a weights array with three elements: [bias, w1, w2].
Augmented Feature Vector
To simplify computation, prepend a 1 to each feature vector: x_augmented = [1, x1, x2]. This allows the bias term to be treated as a regular weight, so the dot product w·x_augmented automatically includes the bias contribution.
Prediction and Update Loop
For each epoch, randomly select a point, compute the dot product of weights and augmented features, predict 1 if ≥ 0 else 0, then update weights using the formula w_new = w_old + learning_rate * (true_label - prediction) * x_augmented. Repeat for 1000 epochs or until convergence.
Converting Weights Back to Line Equation
After training, extract the learned weights [c, w1, w2]. The decision boundary line is w1*x + w2*y + c = 0. To plot it, rearrange to y = -(w1*x + c) / w2, giving slope m = -w1/w2 and y-intercept b = -c/w2.
Convergence & Behavior
Convergence Condition
The perceptron trick converges when no misclassified points remain in the training set, or when a maximum number of epochs is reached. Once convergence occurs, the loop stops and the final weights define the decision boundary.
Line Movement During Training
Initially, the line may be positioned arbitrarily (e.g., horizontally). As misclassified points are encountered, the line rotates and shifts toward them. The line remains stationary when a correctly classified point is sampled, but moves whenever a misclassified point is found.
Notable quotes
This data is linearly separable, which means you can classify this data by drawing a line into these two classes. — Nitesh
Learning rate is generally a small number. You multiply all coordinates with the learning rate and then subtract from the old coefficient. — Nitesh
The code is very simple. For each epoch, randomly select a point and update the coefficient using this rule. That's all I need. — Nitesh
Action items
- Implement the perceptron trick algorithm: initialize weights to zero, loop 1000 times, randomly select a training point, compute prediction, and update weights using w_new = w_old + learning_rate * y * x.
- Augment your feature vectors by prepending 1 to each point (e.g., [1, x1, x2]) so the bias term is treated as a regular weight.
- Use a small learning rate (e.g., 0.01) to ensure stable convergence and avoid overshooting the optimal boundary.
- Test your trained model on the training data to verify that all or most points are correctly classified.
- Extract the final weights and convert them back to a line equation (w1*x + w2*y + c = 0) to visualize the decision boundary.