Neural Networks Explained: From Pixels to Digits
A neural network is a layered mathematical function with thousands of adjustable weights and biases that learns to recognize patterns. Input neurons represent pixel values, hidden layers detect edges and features, and output neurons represent digit predictions. The network's structure mirrors hierarchical pattern recognition: pixels combine into edges, edges into shapes, shapes into complete digits.
Why Neural Networks Matter
The Recognition Problem
Humans effortlessly recognize handwritten digits despite pixel-level variations, yet writing a program to do the same is dauntingly difficult. This gap between biological intuition and computational challenge motivates neural networks as a solution.
Learning vs. Structure
This video focuses on neural network structure—what the network is. A follow-up video will address learning—how it adjusts its parameters to solve problems. Understanding structure is the necessary prerequisite.
Network Architecture
What a Neuron Is
A neuron is simply a container holding a number between 0 and 1, called its activation. This number represents how 'lit up' or active the neuron is, loosely analogous to biological neurons firing.
Layer Structure for Digit Recognition
The network has four layers: an input layer with 784 neurons (one per pixel in a 28×28 image), two hidden layers with 16 neurons each, and an output layer with 10 neurons (one per digit 0–9). Activations in one layer determine activations in the next.
Hierarchical Feature Detection
The hidden layers are expected to detect increasingly abstract features: the first hidden layer detects edges, the second detects patterns like loops or lines, and the output layer combines these to classify digits. This mirrors how humans decompose digit recognition into subcomponents.
How Layers Connect: Weights and Biases
Weights Define Pattern Sensitivity
Each connection from a neuron in one layer to a neuron in the next has a weight (a number). Weights are organized as a matrix where positive weights highlight important pixels and negative weights suppress surrounding pixels. This allows a neuron to detect specific pixel patterns, like edges in a particular region.
Biases Set Activation Thresholds
Each neuron also has a bias, a number added to the weighted sum before activation. The bias determines how high the weighted sum must be before the neuron becomes meaningfully active, effectively raising or lowering the activation threshold.
The Sigmoid Squishification Function
The weighted sum (plus bias) can be any number, but activations must be between 0 and 1. The sigmoid function squishes the real number line into this range: very negative inputs approach 0, positive inputs approach 1, and it transitions smoothly around 0.
Computing One Neuron's Activation
To find a neuron's activation: multiply each input neuron's activation by its corresponding weight, sum all these products, add the bias, then apply the sigmoid function. This single computation happens for every neuron in every hidden and output layer.
Network Parameters and Complexity
Total Parameters in This Network
The network has approximately 13,000 weights and biases combined. From input to first hidden layer: 784 × 16 = 12,544 weights plus 16 biases. Additional connections between hidden layers and to the output layer add more parameters. Each parameter is a 'knob or dial' that can be adjusted.
Why Complexity Is Necessary
The network's complexity—involving thousands of parameters, multiple matrix-vector products, and sigmoid functions—is reassuring. If it were simpler, it would lack the expressiveness to capture the intricate patterns needed to recognize handwritten digits.
The Network as a Function
Network as Composed Function
The entire network is a single complicated function: it takes 784 input numbers (pixel values) and outputs 10 numbers (confidence scores for each digit). Each neuron is itself a function combining outputs from the previous layer. The network chains these functions together across layers.
Matrix Notation for Efficiency
Instead of writing out individual weighted sums, weights are organized into matrices and activations into vectors. A single matrix-vector product plus bias vector plus sigmoid represents an entire layer's computation. This compact notation makes code simpler and faster because libraries heavily optimize matrix multiplication.
What Learning Means
Learning is the process of finding the right values for all 13,000 weights and biases so the network solves the digit-recognition task. The network structure is fixed; learning adjusts the parameters to make the network behave correctly on real data.
Modern Variants: Beyond Sigmoid
ReLU Replaces Sigmoid in Modern Networks
Early networks used sigmoid to squish activations, motivated by biological neuron behavior. Modern networks typically use ReLU (rectified linear unit), which is simpler: it outputs zero if the input is negative, otherwise outputs the input itself. ReLU is easier to train and works better for very deep networks.
Notable quotes
Something in that crazy-smart visual cortex of yours resolves these as representing the same idea. — 3Blue1Brown
All said and done, this network has almost exactly 13,000 total weights and biases. — 3Blue1Brown
The entire network is just a function, one that takes in 784 numbers as an input and spits out 10 numbers as an output. — 3Blue1Brown