Residual Squares

Learning goals

Interpret residuals and squared residuals.
Fit line and quadratic models by minimizing \(\sum_i r_i^2\).
Recognize least squares as projection onto a model column space.
Use \(X^\top\mathbf r=\mathbf0\) to connect residual orthogonality, the normal equations, and a zero gradient.
Center predictors (\(z_i=x_i-\bar x\)) to make columns orthogonal and decouple the normal equations.
Interpret \(\mathbf 1\), \(\mathbf x\), and \(\mathbf x^2\) as feature columns for polynomial regression.

Least-squares cheatsheet

Model: \(\hat y_i=b+mx_i\)
Centered: \(z_i=x_i-\bar x\), \(\hat y_i=h+mz_i\)
Same line; \(h=b+m\bar x\) is its height at \(\bar x\)
Residual: \(r_i=y_i-\hat y_i\)
Goal: Minimize \(\mathrm{SSE}(b,m)=\sum_i r_i^2\)
Picture: Square side \(=|r_i|\)
Area \(=r_i^2\)
Quadratic: Add \(x_i^2\) to the model

Design: \(X=[\,\mathbf{1}\;\mathbf{x}\,],\,\beta=(b,m)\)
Prediction: \(\hat{\mathbf{y}}=X\beta\)
Best prediction: \(\hat{\mathbf{y}}=\mathrm{proj}_{\mathrm{col}(X)}(\mathbf{y})\)
Normal equations: \(X^\top\mathbf{r}=\mathbf{0}\)
For a line: \(\sum_i r_i=0,\quad \sum_i x_i r_i=0\)
Centered columns: \(\mathbf z=\mathbf x-\bar x\mathbf 1\perp\mathbf 1\), so the normal equations decouple:
\(h^*=\bar y,\quad m^*=\sum_i z_iy_i\,/\,\sum_i z_i^2\)
For quadratic: Also require \(\sum_i x_i^2r_i=0\)

Consequence: the least-squares line passes through \((\bar x,\bar y)\).

How to play

Move the pink model.
Each vertical miss \(|r_i|\) is the side of a blue square, so its area is \(r_i^2\).
Shrink the total area \(\sum_i r_i^2\).