Linear Algebra

Linear algebra is the study of vectors, vector spaces, and linear transformations. It is fundamental to modern mathematics, physics, computer science, and machine learning. The subject provides the mathematical framework for understanding systems of linear equations, geometric transformations, and data analysis techniques.

Vectors

Vector Operations

Addition and Scalar Multiplication
Vectors can be added component-wise and multiplied by scalars:

• $(a_1,a_2)+(b_1,b_2)=(a_1+b_1,a_2+b_2)$
• $c(a_1,a_2)=(c{a}_1,c{a}_2)$

Dot Product (Inner Product)
The dot product of two vectors measures their alignment:
$\mathbf{a}\cdot \mathbf{b}=a_1{b}_1+a_2{b}_2+\cdots +a_n{b}_n=|\mathbf{a}||\mathbf{b}|\cos \theta$

Cross Product (in ℝ³)
The cross product produces a vector perpendicular to both inputs:
$\mathbf{a}\times \mathbf{b}=(a_2{b}_3-a_3{b}_2,a_3{b}_1-a_1{b}_3,a_1{b}_2-a_2{b}_1)$

Geometric Interpretation

• Vectors represent direction and magnitude
• Addition follows the parallelogram rule
• Dot product relates to projection
• Cross product gives the area of the parallelogram spanned by two vectors

Vector Spaces

Definition and Axioms
A vector space $V$ over a field $F$ is a set with two operations (addition and scalar multiplication) satisfying:

1. Closure under addition and scalar multiplication
2. Associativity and commutativity of addition
3. Existence of zero vector and additive inverses
4. Distributive and associative properties of scalar multiplication
5. Multiplicative identity: $1\cdot \mathbf{v}=\mathbf{v}$

Subspaces
A subset $W\subseteq V$ is a subspace if it is closed under addition and scalar multiplication, and contains the zero vector.

Span and Linear Combinations
The span of a set of vectors $\{{\mathbf{v}}_1,\dots ,{\mathbf{v}}_k\}$ is the set of all linear combinations:
$\text{span}({\mathbf{v}}_1,\dots ,{\mathbf{v}}_k)=\{c_1{\mathbf{v}}_1+\cdots +c_k{\mathbf{v}}_k:c_i\in F\}$

Linear Independence
Vectors $\{{\mathbf{v}}_1,\dots ,{\mathbf{v}}_k\}$ are linearly independent if the only solution to $c_1{\mathbf{v}}_1+\cdots +c_k{\mathbf{v}}_k=0$ is $c_1=\cdots =c_k=0$.

Basis and Dimension
A basis is a linearly independent spanning set. The dimension of a vector space is the number of vectors in any basis.

Matrices

Matrix Operations

Addition and Scalar Multiplication
Matrices of the same dimensions can be added entry-wise. Scalar multiplication multiplies each entry.

Matrix Multiplication
For matrices $A$ (m×n) and $B$ (n×p), the product $AB$ is m×p with:
$(AB{)}_{ij}={\sum }_{k=1}^n{A}_{ik}{B}_{kj}$

Transpose
The transpose $A^T$ swaps rows and columns: $(A^T{)}_{ij}=A_{ji}$

Trace
The trace of a square matrix is the sum of its diagonal entries:
$\text{tr}(A)={\sum }_{i=1}^n{A}_{ii}$

Special Matrices

Type	Definition
Identity	$I_{ij}=1$ if $i=j$, else 0
Diagonal	Non-zero entries only on main diagonal
Upper triangular	Zero entries below main diagonal
Lower triangular	Zero entries above main diagonal
Symmetric	$A=A^T$
Skew-symmetric	$A=-A^T$
Orthogonal	$A^{T}A=A{A}^T=I$

Matrix Properties

Rank
The rank of a matrix is the dimension of its column space (or equivalently, row space).

Inverse
A square matrix $A$ is invertible if there exists $A^{-1}$ such that $A{A}^{-1}=A^{-1}A=I$. A matrix is invertible if and only if its determinant is non-zero.

Determinants
The determinant is a scalar value encoding properties of a matrix:

• For 2×2: $\det \left(\begin{array}{cc}a& b\\ c& d\end{array}\right)=ad-bc$
• $\det (AB)=\det (A)\det (B)$
• $\det (A^T)=\det (A)$
• $\det (A^{-1})=1/\det (A)$

Systems of Linear Equations

Solving Systems

Gaussian Elimination
A systematic method for solving linear systems by applying row operations:

1. Swap two rows
2. Multiply a row by a non-zero scalar
3. Add a multiple of one row to another

Row Echelon Form (REF)
A matrix is in REF if:

• All zero rows are at the bottom
• The leading entry of each non-zero row is to the right of the leading entry above it

Reduced Row Echelon Form (RREF)
RREF additionally requires:

• Leading entries are all 1
• Leading entries are the only non-zero entry in their column

LU Decomposition
Express a matrix as $A=LU$ where $L$ is lower triangular and $U$ is upper triangular. Useful for solving multiple systems with the same coefficient matrix.

Solution Types

For a system $A\mathbf{x}=\mathbf{b}$:

• Unique solution: $\text{rank}(A)=\text{rank}([A|\mathbf{b}])=n$ (number of unknowns)
• Infinite solutions: $\text{rank}(A)=\text{rank}([A|\mathbf{b}])<n$
• No solution: $\text{rank}(A)<\text{rank}([A|\mathbf{b}])$

Geometric Interpretation

• In ℝ², each equation represents a line; solutions are intersections
• In ℝ³, each equation represents a plane; solutions are intersections

Linear Transformations

Definitions

A function $T:V\to W$ between vector spaces is a linear transformation if:

• $T(\mathbf{u}+\mathbf{v})=T(\mathbf{u})+T(\mathbf{v})$
• $T(c\mathbf{v})=cT(\mathbf{v})$

Kernel (Null Space)
$\ker (T)=\{\mathbf{v}\in V:T(\mathbf{v})=0\}$

Image (Range)
$\text{im}(T)=\{T(\mathbf{v}):\mathbf{v}\in V\}$

Rank-Nullity Theorem
$\dim (V)=\dim (\ker (T))+\dim (\text{im}(T))$

Matrix Representation

Every linear transformation between finite-dimensional vector spaces can be represented by a matrix. The matrix depends on the choice of bases for the domain and codomain.

Change of Basis
If $P$ is the change of basis matrix from basis $B$ to basis $B^{\prime }$, then the matrix representation transforms as:
$A^{\prime }=P^{-1}AP$

Eigenvalues and Eigenvectors

Definitions

An eigenvector $\mathbf{v}$ of a matrix $A$ satisfies:
$A\mathbf{v}=\lambda \mathbf{v}$
where $\lambda$ is the corresponding eigenvalue.

Characteristic Polynomial
Eigenvalues are roots of the characteristic polynomial:
$p(\lambda )=\det (A-\lambda I)$

Finding Eigenvalues and Eigenvectors

1. Solve $\det (A-\lambda I)=0$ for eigenvalues
2. For each eigenvalue, solve $(A-\lambda I)\mathbf{v}=0$ for eigenvectors

Diagonalisation

A matrix $A$ is diagonalisable if $A=PD{P}^{-1}$ where $D$ is diagonal and $P$ contains eigenvectors as columns.

When is a Matrix Diagonalisable?

• A matrix is diagonalisable if it has $n$ linearly independent eigenvectors
• Matrices with distinct eigenvalues are always diagonalisable

Spectral Theorem
Every real symmetric matrix is orthogonally diagonalisable: $A=PD{P}^T$ where $P$ is orthogonal.

Applications

• Principal Component Analysis (PCA): Dimensionality reduction using eigenvectors of covariance matrices
• Differential Equations: Solving systems ${\mathbf{x}}^{\prime }=A\mathbf{x}$ via diagonalisation
• Markov Chains: Steady-state distributions from eigenvectors
• Google PageRank: Web page importance via eigenvector of link matrix

Inner Product Spaces

Inner Products
An inner product $⟨\cdot ,\cdot ⟩$ on a vector space satisfies:

• Linearity in the first argument
• Conjugate symmetry: $⟨\mathbf{u},\mathbf{v}⟩=\overline{⟨\mathbf{v},\mathbf{u}⟩}$
• Positive definiteness: $⟨\mathbf{v},\mathbf{v}⟩\ge 0$, with equality iff $\mathbf{v}=0$

Norms and Distances
The norm induced by an inner product: $\Vert \mathbf{v}\Vert =\sqrt{⟨\mathbf{v},\mathbf{v}⟩}$
Distance: $d(\mathbf{u},\mathbf{v})=\Vert \mathbf{u}-\mathbf{v}\Vert$

Orthogonality
Vectors $\mathbf{u}$ and $\mathbf{v}$ are orthogonal if $⟨\mathbf{u},\mathbf{v}⟩=0$.

Gram-Schmidt Orthogonalisation
An algorithm to convert a basis into an orthonormal basis:

1. ${\mathbf{e}}_1={\mathbf{v}}_1/\Vert {\mathbf{v}}_1\Vert$
2. ${\mathbf{e}}_k=({\mathbf{v}}_k-{\sum }_{j<k}⟨{\mathbf{v}}_k,{\mathbf{e}}_j⟩{\mathbf{e}}_j)/\Vert \cdot \Vert$

QR Decomposition
Express $A=QR$ where $Q$ is orthogonal and $R$ is upper triangular. Obtained via Gram-Schmidt on the columns of $A$.

Singular Value Decomposition (SVD)

Every matrix $A$ (m×n) can be decomposed as:
$A=U\Sigma V^T$
where:

• $U$ (m×m) is orthogonal (left singular vectors)
• $\Sigma$ (m×n) is diagonal (singular values)
• $V$ (n×n) is orthogonal (right singular vectors)

Applications

• Data compression: Low-rank approximations
• Recommender systems: Matrix factorisation for collaborative filtering
• Image processing: Noise reduction and compression
• Pseudoinverse: Computing $A^{+}$ for least squares problems

Glossary

Term	Definition
Span	Set of all linear combinations of a set of vectors
Basis	Linearly independent spanning set
Rank	Dimension of the column space
Nullity	Dimension of the null space
Kernel	Solution set of $A\mathbf{x}=0$
Eigenvalue	Scalar $\lambda$ where $A\mathbf{v}=\lambda \mathbf{v}$
Eigenvector	Non-zero vector $\mathbf{v}$ where $A\mathbf{v}=\lambda \mathbf{v}$
Orthogonal	Perpendicular; inner product equals zero
Singular value	Square root of eigenvalue of $A^{T}A$

Tools

• Matrix Calculator — Online tool for matrix operations
• NumPy — Python library for numerical computing
• MATLAB — Commercial numerical computing environment
• Interactive Linear Algebra — Georgia Tech’s interactive textbook
• Symbolab — Step-by-step matrix solver

Learning Resources

Books

• Introduction to Linear Algebra by Gilbert Strang — Classic textbook with geometric intuition
• Linear Algebra Done Right by Sheldon Axler — Proof-focused, avoids determinants early
• Linear Algebra and Its Applications by David Lay — Applications-oriented approach
• Matrix Analysis by Horn and Johnson — Advanced reference

Courses

• MIT 18.06 Linear Algebra — Gilbert Strang’s famous course
• 3Blue1Brown: Essence of Linear Algebra — Visual, intuitive explanations
• Khan Academy: Linear Algebra — Comprehensive free course

See Also

• Calculus — Multivariable calculus uses linear algebra extensively
• Abstract Algebra — Generalises vector space concepts to modules and rings
• Machine Learning — Linear algebra is the computational backbone of ML algorithms