Orthogonal Projections & Least Squares Practice Questions with Answers and a Guided Lesson
Use the question set below to practice orthogonal projections and least squares: closest vectors in subspaces, decompositions \(v=p+r\) with \(p\in S\) and \(r\perp S\), projection onto a line, projection matrices with \(P^2=P\) and \(P^T=P\), column-space formulas such as \(A(A^TA)^{-1}A^T\) when \(A\) has full column rank, normal equations \(A^TAx=A^Tb\), residual orthogonality, best constant fits, and what changes when \(A^TA\) is singular. Open the lesson for concise worked examples and quick checks.
Answer the question set and review your mistakes at the end.
How this projection and least-squares practice works
1. Take the practice set: answer questions about projections, residuals, projection matrices, normal equations, and best fits.
2. Open the lesson: review the formulas, recognition tests, worked examples, and single-answer checks.
3. Retry: return to the question set and first decide whether the problem asks for a closest vector, a projection matrix, a residual condition, or a least-squares coefficient.
What you will learn in the orthogonal projections and least squares lesson
Orthogonal projection geometry
Closest vector: \(\operatorname{proj}_S(v)\) is the unique point of \(S\) closest to \(v\)
Purpose: Build a reliable workflow for projections and least squares: decompose a vector into subspace plus orthogonal error, compute projections onto lines and column spaces, recognize orthogonal projection matrices, solve normal equations, and know what remains true when the columns of \(A\) are dependent.
Success criteria
State the orthogonal decomposition \(v=p+r\), with \(p\in S\) and \(r\perp S\).
Use the line formula \(\operatorname{proj}_{\operatorname{span}(u)}(v)=\dfrac{v\cdot u}{u\cdot u}u\).
Recognize that an orthogonal projection matrix satisfies \(P^2=P\) and \(P^T=P\).
Use \(QQ^T\) for a subspace with orthonormal basis columns \(Q\), and \(Q^Tb\) for the least-squares coefficients.
Use \(A(A^TA)^{-1}A^T\) when \(A\) has full column rank.
Derive least squares from residual orthogonality \(A^T(b-A\hat{x})=0\).
Separate the fitted vector \(A\hat{x}\) from the coefficient vector \(\hat{x}\).
Handle rank-deficient cases without assuming \(A^TA\) is invertible.
Key vocabulary
Projection: the closest vector in a subspace.
Residual: the error \(r=v-p\) or \(r=b-A\hat{x}\).
Orthogonal complement: \(S^\perp=\{w:w\cdot s=0\text{ for all }s\in S\}\).
Projection matrix: a linear map \(P\) with \(P^2=P\); orthogonal projections also satisfy \(P^T=P\).
Column space: all vectors \(Ax\), the possible fitted outputs of \(A\).
Normal equations: \(A^TA\hat{x}=A^Tb\), equivalent to residual orthogonality.
Quick pre-check
Pre-check: If \(p=\operatorname{proj}_S(v)\), what is true about \(v-p\)?
Hint: The closest-point condition is equivalent to the error being perpendicular to every direction in the subspace.
Projection means subspace part plus perpendicular error
Learning goal: Recognize the geometry behind every projection and least-squares computation.
Key idea
For a subspace \(S\) of an inner product space, every vector \(v\) can be split as \[v=p+r,\qquad p\in S,\quad r\in S^\perp.\] The vector \(p\) is \(\operatorname{proj}_S(v)\), and \(r\) is the residual. The Pythagorean theorem gives \(\|v-s\|^2=\|r\|^2+\|p-s\|^2\) for every \(s\in S\), so \(p\) is the closest vector in \(S\).
Recognition checklist
Identify the target subspace \(S\).
Find a candidate \(p\in S\).
Check \(v-p\perp S\), usually by dotting with a basis of \(S\).
If both conditions hold, \(p\) is the projection.
Worked example
Example: Project \(v=(1,0)\) onto the line \(S=\operatorname{span}((1,1))\).
A vector on the line has form \(p=t(1,1)\). The error \(v-p=(1-t,-t)\) must be orthogonal to \((1,1)\), so \((1-t)+(-t)=0\). Thus \(1-2t=0\), so \(t=\tfrac12\) and \(p=(\tfrac12,\tfrac12)\).
Try it
Try it: What is the projection of \((1,0)\) onto \(\operatorname{span}((1,1))\)?
Hint: The coefficient is \(\dfrac{(1,0)\cdot(1,1)}{(1,1)\cdot(1,1)}=\dfrac12\).
Use one dot-product coefficient
Learning goal: Compute line projections quickly without accidentally assuming the direction vector has unit length.
Key idea
If \(u≠0\), the projection of \(v\) onto \(\operatorname{span}(u)\) is \[\operatorname{proj}_{u}(v)=\frac{v\cdot u}{u\cdot u}u.\] If \(u\) is a unit vector, this simplifies to \((v\cdot u)u\).
Formula notes
The denominator is \(u\cdot u=\|u\|^2\), not just \(\|u\|\).
If \(v\) is already on the line, the projection is \(v\).
If \(v\perp u\), the projection is \(0\).
The residual \(v-\operatorname{proj}_u(v)\) is perpendicular to \(u\).
The coefficient is \(\dfrac{(2,0)\cdot(1,1)}{(1,1)\cdot(1,1)}=\dfrac{2}{2}=1\). Therefore the projection is \(1(1,1)=(1,1)\).
Try it
Try it: What is the projection of \((2,0)\) onto the line spanned by \((1,1)\)?
Hint: The dot product with \((1,1)\) is \(2\), and \((1,1)\cdot(1,1)=2\).
Orthogonal projection matrices are idempotent and symmetric
Learning goal: Recognize projection matrices from algebraic tests and read their range, kernel, rank, and trace.
Key idea
A projection satisfies \(P^2=P\): once a vector is projected, projecting again changes nothing. It is an orthogonal projection exactly when the residual is perpendicular to the range; in standard coordinates this means \(P^T=P\).
Matrix tests
\(P^2=P\): projection or idempotent map.
\(P^T=P\): orthogonal projection in Euclidean space.
\(\operatorname{Range}(P)\) is the target subspace, and eigenvalue \(1\) corresponds to vectors in the range.
\(\ker P=S^\perp\) for an orthogonal projection onto \(S\), and \(I-P\) projects onto \(S^\perp\).
Eigenvalues are only \(0\) and \(1\), so \(\operatorname{tr}P=\operatorname{rank}P\).
Worked example
Example: The projection onto the \(x\)-axis sends \((x,y)\) to \((x,0)\). What properties does its matrix have?
The matrix is \(\operatorname{diag}(1,0)\). Applying it twice gives the same matrix, so \(P^2=P\). It equals its transpose, so \(P^T=P\). Its range is the \(x\)-axis and its kernel is the \(y\)-axis.
Try it
Try it: Which identities hold for every orthogonal projection matrix \(P\)?
Hint: Projecting twice changes nothing, and orthogonal projection matrices are symmetric.
Projecting onto \(\operatorname{Col}(A)\)
Learning goal: Choose the correct projection formula from the columns you have.
Key idea
If \(Q\) has orthonormal columns spanning \(S\), then the projection matrix onto \(S\) is \(P=QQ^T\). If \(A\) has full column rank but not necessarily orthonormal columns, then \[P=A(A^TA)^{-1}A^T.\] Both formulas produce the closest vector in \(\operatorname{Col}(A)\).
Column-space steps
If the columns are orthonormal, use \(QQ^T\) for the projection matrix.
For least-squares coefficients with orthonormal columns \(Q\), use \(\hat{x}=Q^Tb\).
If the columns are independent but not orthonormal, use \(A(A^TA)^{-1}A^T\).
If columns are dependent, reduce to an independent basis or use the pseudoinverse.
Always check the output lies in the column space and the residual is orthogonal to the column space.
Worked example
Example: Let \(Q\) have the single unit column \(q=(1/\sqrt2,1/\sqrt2)\). What does \(QQ^T\) do?
It projects any vector \((a,b)\) onto the line \(y=x\), giving \(\left(\dfrac{a+b}{2},\dfrac{a+b}{2}\right)\). For example, \((2,0)\) is sent to \((1,1)\).
Try it
Try it: If \(Q\) has orthonormal columns, what is the projection matrix onto \(\operatorname{Col}(Q)\)?
Hint: Orthonormal columns make \(Q^TQ=I\), so the full formula collapses.
Least squares is projection onto a column space
Learning goal: Translate an inconsistent system \(Ax\approx b\) into a projection problem.
Key idea
A least-squares solution minimizes \(\|b-Ax\|\). The fitted vector \(A\hat{x}\) is the projection of \(b\) onto \(\operatorname{Col}(A)\), so the residual \(r=b-A\hat{x}\) is orthogonal to every column of \(A\). This is exactly \(A^Tr=0\), or \(A^TA\hat{x}=A^Tb\).
Normal-equation facts
The normal equations are \(A^TA\hat{x}=A^Tb\).
If \(A\) has full column rank, \(A^TA\) is invertible and the coefficient vector \(\hat{x}\) is unique.
If \(A=Q\) has orthonormal columns, then \(\hat{x}=Q^Tb\).
The fitted vector \(A\hat{x}\) is the projection of \(b\) onto \(\operatorname{Col}(A)\).
The residual is perpendicular to every column of \(A\).
Worked example
Example: Find the best constant fit to the data \(1,3,5\).
Fitting a constant \(c\) means projecting \((1,3,5)\) onto the line spanned by \((1,1,1)\). The coefficient is the average \(c=(1+3+5)/3=3\). The residual is \((-2,0,2)\), whose entries sum to \(0\), so it is orthogonal to the constant direction.
Try it
Try it: In least squares, the residual \(b-A\hat{x}\) is orthogonal to which space?
Hint: The normal equations say \(A^T(b-A\hat{x})=0\), so each column has zero dot product with the residual.
When coefficients are not unique, the fitted vector still is
Learning goal: Avoid overusing inverse formulas when columns of \(A\) are dependent.
Key idea
If \(A\) does not have full column rank, then \(A^TA\) is singular. There may be many least-squares coefficient vectors \(\hat{x}\), because different coefficients can give the same fitted vector. The geometry still has a unique closest vector \(p\in\operatorname{Col}(A)\), so \(A\hat{x}=p\) is unique.
Rank-deficient facts
\(A^TA\) is invertible exactly when the columns of \(A\) are independent.
Normal equations still characterize least-squares minimizers, but may have multiple solutions.
The projection \(p=A\hat{x}\) is unique.
The pseudoinverse \(A^+b\) gives the least-squares solution with smallest coefficient norm.
Worked example
Example: Suppose both columns of \(A\) are \((1,0)\), and \(b=(2,1)\). What happens?
The column space is the \(x\)-axis, so the fitted vector is the projection \(p=(2,0)\). Any coefficients with \(x_1+x_2=2\) give the same fitted vector, so the coefficients are not unique. The residual is \(b-p=(0,1)\), perpendicular to the \(x\)-axis.
Try it
Try it: If \(A\) is rank deficient, what can still be true for least squares?
Hint: The closest point in a subspace is unique even when several coefficient vectors produce it.
Most mistakes confuse the coefficient vector with the projection
Learning goal: Finish with a compact checklist for common projection and least-squares errors.
Common traps
Nonunit direction: divide by \(u\cdot u\) when projecting onto \(\operatorname{span}(u)\).
Projection vs. residual: \(p\in S\), while \(v-p\in S^\perp\).
Projection matrix: \(P^2=P\) alone is not enough for orthogonal projection; also check symmetry.
Least squares: \(A\hat{x}\) is the projection of \(b\), not necessarily \(b\) itself.
Rank deficiency: \(\hat{x}\) may not be unique, but \(A\hat{x}\) is unique.
Worked example
Example: If \(v=(2,2)\) already lies on the line \(y=x\), what is its projection onto that line?
The projection is \(v\) itself: \((2,2)\). A vector already in the subspace has zero residual, and zero is orthogonal to every subspace.
Try it
Try it: What is the best constant least-squares fit for the data \(1,3,5\)?
Hint: A best constant fit is the average of the data values.
Final recap
Projection means \(v=p+r\) with \(p\in S\) and \(r\perp S\).
For a line, use \(\dfrac{v\cdot u}{u\cdot u}u\).
Orthogonal projection matrices satisfy \(P^2=P\) and \(P^T=P\).
\(\ker P=S^\perp\) for projection onto \(S\), and \(I-P\) projects onto \(S^\perp\).
For orthonormal columns \(Q\), the projection matrix is \(QQ^T\) and the least-squares coefficients are \(Q^Tb\).
For full column rank \(A\), the column-space projection matrix is \(A(A^TA)^{-1}A^T\).
Least squares makes \(b-A\hat{x}\) orthogonal to \(\operatorname{Col}(A)\).
Normal equations are \(A^TA\hat{x}=A^Tb\).
Rank deficiency can make \(\hat{x}\) nonunique, while \(A\hat{x}\) remains the unique projection.
Next step: Close this lesson and try the quiz again. For each problem, first identify the target subspace, then decide whether you need a vector projection, a projection matrix, or a least-squares residual condition.
