One factorization for every matrix
Eigenvalues only work cleanly for some square matrices. The singular value decomposition works for every matrix, of any shape, with no exceptions: it writes A as three simple pieces, A = U Σ Vᵀ.
Geometrically the message is beautiful: whatever a matrix does, it is only ever a rotation, then a stretch along perpendicular axes, then another rotation. No matrix does anything more complicated than that.
What the three pieces are
Vᵀ picks out a set of perpendicular input directions and rotates them onto the axes. Σ is diagonal and non-negative: it stretches each axis by a singular value σ₁ ≥ σ₂ ≥ … ≥ 0. U rotates the stretched result into its final perpendicular output directions.
So V holds the special input directions, U the matching output directions, and Σ says how much each is stretched. The singular values are the sizes that matter, always listed largest first.
Any matrix you build with the sliders is secretly a rotation, a stretch along two perpendicular axes, and another rotation. The two stretch factors are its singular values; when one hits zero the matrix is singular and squashes the plane to a line.
Singular values, rank and importance
The number of non-zero singular values is the rank of the matrix — a far more reliable way to measure rank on real, noisy data than counting pivots. And because they are ordered, the first few singular values capture the directions where the matrix does the most; the tiny ones are often just noise.
The singular values are tied to eigenvalues under the hood: they are the square roots of the eigenvalues of AᵀA, which is why the SVD always exists even when A itself has none.
Why the SVD runs modern data science
Keep only the largest few singular values and you get the best possible low-rank approximation of a matrix — the mathematical heart of image and data compression, noise removal, and recommender systems.
It is also exactly what Principal Component Analysis computes: the top singular directions are the axes along which data varies most. From compressing photos to reducing the dimensions of a dataset before machine learning, the SVD is arguably the single most useful factorization in applied mathematics.
Low-rank approximation in one line
Throw away the small singular values and rebuild A from the large ones. You store a fraction of the numbers and lose almost nothing that mattered — that is compression, denoising and PCA, all from the same idea.
A worked example
How would you find the singular values of a matrix A, and what would a singular value of 0 tell you?
- Form AᵀA and find its eigenvalues.AᵀA is square and symmetric, so it always has real, non-negative eigenvalues.
- Take the square roots: σᵢ = √(eigenvalue). These are the singular values, ordered largest first.The singular values of A are the square roots of the eigenvalues of AᵀA.
- A singular value of 0 means that input direction is completely flattened.Zero stretch on an axis means the matrix collapses that direction — it is part of the nullspace.
Singular values are √(eigenvalues of AᵀA); a zero among them signals a flattened direction, so the count of non-zero ones is the rank.