Two vectors are orthogonal exactly when:

Their dot product is zero. A zero dot product is the definition of a right angle between vectors.

The projection of b onto the line through a is:

(a·b / a·a)·a. proj = (a·b / a·a)·a — the multiple of a closest to b.

Least squares is used when Ax = b:

Has no exact solution. When b lies outside the column space there is no exact solution, so we find the closest x̂ instead.

The normal equations for the least-squares solution are:

AᵀA x̂ = Aᵀb. Setting the error orthogonal to the columns gives AᵀA x̂ = Aᵀb.

In a projection, the error vector (b − proj) is always:

Perpendicular to the subspace. The closest point leaves an error orthogonal to the subspace — the whole basis of least squares.

Gram–Schmidt produces:

An orthonormal basis. It converts any basis into a mutually perpendicular, unit-length (orthonormal) basis.

Orthogonality, Projections & Least Squares

Orthogonality is a right angle, measured by the dot product

Two vectors are orthogonal — perpendicular — exactly when their dot product is zero. This single number replaces protractors: no matter how many dimensions, a·b = 0 means a right angle.

A set of vectors that are all mutually perpendicular is an orthogonal set; if each also has length 1, it is orthonormal. Orthonormal sets are the nicest bases there are — coordinates become simple dot products, and no direction interferes with another.

Projection: the closest point in a subspace

The projection of a vector b onto a line through a is its shadow on that line — the point on the line nearest to b. The leftover, b minus its shadow, is perpendicular to the line, and that "error is orthogonal" fact is the key to everything that follows.

proj_a(b) = (a·b / a·a) · athe multiple of a closest to b; b − proj is perpendicular to a

Least squares: the best answer to an unsolvable system

Real data rarely lands exactly on a line, so Ax = b often has no solution — b lies outside the column space. Rather than give up, we find the x̂ that comes closest, minimising the length of the error. That is least squares, and it is how you fit a line or curve to measurements.

The trick is projection: the best b we can reach is the projection of the real b onto the column space, and the error is orthogonal to that space. Writing "error ⟂ columns" gives the normal equations.

The normal equations

The least-squares solution solves AᵀA x̂ = Aᵀb. It is the workhorse behind linear regression and curve-fitting — the same formula whether you are fitting two points or two million.

Orthonormal bases and Gram–Schmidt

Because orthonormal bases make projection and least squares trivial, it is worth being able to build one. Gram–Schmidt takes any basis and straightens it into an orthonormal one: keep the first vector, subtract from the second its projection onto the first (leaving the perpendicular part), and repeat, normalising as you go.

The result is written as the QR factorization, A = QR, with Q orthonormal — a standard first step in numerical least squares.

A worked example

Worked example

Project b = [2, 3] onto the line through a = [1, 0]. What is the error, and is it perpendicular to a?

a·b = (1)(2) + (0)(3) = 2, and a·a = 1.Plug into the projection formula.
proj = (2/1)·[1, 0] = [2, 0].The shadow of b on the horizontal line.
Error = b − proj = [2, 3] − [2, 0] = [0, 3]; and a·[0, 3] = 0.The leftover is vertical, perpendicular to the horizontal line — as projection guarantees.

Projection [2, 0], error [0, 3], and the error is orthogonal to a — the defining property that makes least squares work.