Least Squares Finds the Closest Model

No straight line will usually pass through every noisy observation. Least squares replaces the demand for a perfect match with a precise compromise: choose the parameters whose predictions have the smallest total squared discrepancy.

The objective

For the line $\hat y_i=\beta_0+\beta_1x_i$, define

$$S(\beta_0,\beta_1)=\sum_{i=1}^{n}\left[y_i-(\beta_0+\beta_1x_i)\right]^2.$$

The least-squares estimates $\hat\beta_0$ and $\hat\beta_1$ minimize $S$.

The fitted line

For ordinary simple linear regression,

$$\hat\beta_1=\frac{\sum (x_i-\bar x)(y_i-\bar y)}{\sum (x_i-\bar x)^2}, \qquad \hat\beta_0=\bar y-\hat\beta_1\bar x.$$

The slope is large when $x$ and $y$ vary together relative to the variation in $x$. The intercept places the fitted line through $(\bar x,\bar y)$.

The matrix view

Many linear models can be written as

$$\mathbf y=X\boldsymbol\beta+\boldsymbol\varepsilon.$$

Least squares minimizes $\|\mathbf y-X\boldsymbol\beta\|^2$. Its solution satisfies the normal equations

$$X^TX\hat{\boldsymbol\beta}=X^T\mathbf y.$$

Geometrically, the fitted vector $X\hat{\boldsymbol\beta}$ is the projection of $\mathbf y$ onto the space of predictions the model can produce.

Assumptions and uncertainty

Least squares can calculate a line without a probability model. Statistical intervals and tests additionally rely on assumptions such as independent errors, constant error variance, and an appropriate mean structure. Normal errors are commonly assumed for exact small-sample inference.

Derive the simple-regression normal equations

Differentiate $S(\beta_0,\beta_1)$ with respect to both coefficients and set the derivatives to zero:

$\sum (y_i-\beta_0-\beta_1x_i)=0,$ $\sum x_i(y_i-\beta_0-\beta_1x_i)=0.$

The first equation says residuals sum to zero; the second says residuals are orthogonal to $x$. Solving gives the familiar formulas for slope and intercept.

Check your understanding

Why does rescaling $x$ from meters to centimeters change the numerical slope but not fitted values?

Show the reasoning

One meter equals 100 centimeters, so a one-centimeter change is 1/100 as large. The slope per centimeter is the slope per meter divided by 100; multiplying by the rescaled $x$ restores the same predictions.