Linear Algebra Fundamentals for Machine Learning
A comprehensive introduction to vectors covering definition, components, magnitude, operations (addition, subtraction, scalar multiplication), the dot product, orthogonality, and proofs of key properties. Extends concepts from 2D to n-dimensional space with applications to finding angles between vectors and geometric proofs.
What is a Vector
Vector Definition: Direction and Magnitude
A vector is defined by two core properties: direction (orientation) and magnitude (length). Vectors are typically represented as arrows on a graph, where the arrow's direction and length encode the vector's information.
Vectors Don't Require Origin Starting Point
Vectors can start anywhere on a plane, not just at the origin. What matters is the direction and length of the arrow; the position is irrelevant. This is called the translational property of vectors.
Vector Components Capture Direction and Magnitude
A vector's direction and magnitude can be captured using components: the X-component (horizontal movement) and Y-component (vertical movement). For example, vector V = (3, 1) means move 3 units right and 1 unit up.
Vector Magnitude and the Pythagorean Theorem
Finding Vector Length in 2D
The length (or magnitude) of a 2D vector is found using the Pythagorean theorem. For vector V = (3, 1), the components form a right triangle where the vector is the hypotenuse. Length = √(3² + 1²) = √10.
Vector Length Examples
Multiple examples demonstrate the formula: U = (-2, 7) has length √(4 + 49) = √53; W = (3, -2) has length √(9 + 4) = √13. The formula works regardless of negative components.
Vector Operations: Addition and Subtraction
Vector Addition: Component-Wise
To add two vectors, add their corresponding components. V = (3, 2) + U = (1, 4) = (4, 6). Geometrically, place the tail of U at the head of V; the resulting vector goes from the common starting point to the final endpoint.
Vector Addition is Commutative
The order of vector addition doesn't matter: V + U = U + V. This can be proven geometrically by the parallelogram rule—both orderings reach the same endpoint.
Scalar Multiplication: Scaling Vectors
Multiplying a vector by a scalar (number) scales its length. 2V doubles the vector's length; 0.5V halves it. Multiplying by -1 reverses direction: -V points opposite to V.
Vector Subtraction: Using Negative Scalars
Vector subtraction is defined as U - V = U + (-V). Geometrically, U - V points from the head of V to the head of U, forming the third side of a triangle.
Extending to Three Dimensions and Beyond
3D Vectors and Graphing
Vectors with three components (X, Y, Z) are graphed in 3D space. For V = (1, 4, 3), move 1 unit in X, 4 in Y, and 3 in Z. Dashed lines help visualize depth. All 2D vector operations (addition, subtraction, scalar multiplication) apply identically.
Vector Length in 3D and Higher Dimensions
The magnitude formula extends naturally: for V = (1, 4, 3), length = √(1² + 4² + 3²) = √26. For n-dimensional vectors, sum the squares of all n components and take the square root.
R^n Notation: Vectors in n-Dimensional Space
R^n denotes the set of all n-tuples of real numbers. V ∈ R² means V is a 2D vector; U ∈ R³ means U is 3D. This notation allows abstract treatment of vectors regardless of dimension, enabling linear algebra to work in any dimension.
Algebraic Properties of Vectors
Vector Addition is Commutative and Associative
Commutativity: U + V = V + U. Associativity: (U + V) + W = U + (V + W). These are proven by applying the commutative and associative properties of real number addition to each component.
The Zero Vector
The zero vector 0 has all components equal to zero. For any vector V, V + 0 = V. This is distinct from the scalar zero; vectors require vector addition.
Proof Strategy: Reduce to Real Numbers
Vector properties are proven by breaking vectors into components (which are real numbers), applying known properties of real number arithmetic, then recombining into vectors. This approach works for any dimension.
The Dot Product
Dot Product Definition
The dot product of U and V is: U · V = u₁v₁ + u₂v₂ + ... + uₙvₙ. Multiply corresponding components and sum. The result is a scalar (single number), not a vector, so it's also called the scalar product.
Dot Product and Vector Length
The length of a vector can be expressed as: ||V|| = √(V · V). This is because V · V = v₁² + v₂² + ... + vₙ², which is exactly what appears under the square root in the magnitude formula.
Dot Product and Angle Between Vectors
The angle θ between two vectors satisfies: cos(θ) = (U · V) / (||U|| × ||V||). This formula connects the dot product to geometry, allowing computation of angles between vectors in any dimension.
Finding Angles Between Vectors
Angle Computation Example
For U = (2, 1, -2) and V = (1, 1, 1): compute U · V = 1, ||U|| = 3, ||V|| = √3. Then cos(θ) = 1/(3√3), so θ = arccos(1/(3√3)) ≈ 1.37 radians ≈ 78.9 degrees.
Orthogonal Vectors
Orthogonality: Perpendicular Vectors
Two vectors U and V are orthogonal (perpendicular) if and only if U · V = 0. This is because cos(90°) = 0, so the angle formula gives U · V = ||U|| × ||V|| × 0 = 0.
Testing Orthogonality Example
For U = (-3, 4) and V = (4, 3): U · V = (-3)(4) + (4)(3) = -12 + 12 = 0. Therefore, U and V are orthogonal. Graphically, they appear to meet at a 90-degree angle.
Parallelogram Diagonals Proof
Theorem: Diagonals Perpendicular iff Sides Equal
For a parallelogram with sides U and V, the diagonals (U + V) and (U - V) are perpendicular if and only if ||U|| = ||V||. This is proven by expanding the dot product (U + V) · (U - V) = ||U||² - ||V||².
Proof Forward Direction
Assume (U + V) · (U - V) = 0. Expanding: ||U||² - 2(U · V) + 2(U · V) - ||V||² = 0, which simplifies to ||U||² = ||V||². Taking square roots (both lengths are positive): ||U|| = ||V||.
Proof Reverse Direction
Assume ||U|| = ||V||. Consider (U + V) · (U - V) = ||U||² - ||V||² = 0 (by substitution). Therefore, the diagonals are orthogonal.
Notable quotes
A vector is really just two pieces of information: direction and length. — Instructor
Vectors need not start at the origin; they can start anywhere. — Instructor
The beauty of linear algebra is we can push these concepts to higher dimensions without needing to graph. — Instructor
Action items
- Practice computing vector components from graphed arrows; verify by reconstructing the arrow from components.
- Calculate magnitudes for 2D and 3D vectors using the Pythagorean theorem formula; check results with a calculator.
- Perform vector addition and subtraction component-wise; verify geometrically using head-to-tail method.
- Compute dot products for given vector pairs and verify orthogonality by checking if dot product equals zero.
- Find angles between vectors using the formula cos(θ) = (U · V) / (||U|| × ||V||); use a calculator for inverse cosine.
- Prove vector addition commutativity and associativity by expanding components and applying real number properties.
- Test the parallelogram diagonal theorem: create vectors with equal lengths and verify diagonals are perpendicular.