Gauss–Newton Linearizes Nonlinear Least Squares

Gauss–Newton solves nonlinear least-squares problems by treating the residuals as approximately linear near the current parameter estimate. It is fast when that local picture is accurate and the starting point is already reasonably close.

Linearize the residuals

Let $\mathbf r(\boldsymbol\theta)$ be the residual vector. Near the current estimate $\boldsymbol\theta_k$,

$$\mathbf r(\boldsymbol\theta_k+\boldsymbol\delta) \approx \mathbf r_k+J_k\boldsymbol\delta,$$

where $J_k$ is the Jacobian matrix with entries

$$(J_k)_{ij}=\frac{\partial r_i}{\partial\theta_j}.$$

Gauss–Newton chooses $\boldsymbol\delta$ to minimize the squared length of this linearized residual.

Why the approximation works

The exact Hessian of the sum of squared residuals contains $J^TJ$ plus terms involving residuals and second derivatives. Gauss–Newton drops those extra terms. Near a good fit, residuals are often small, making $J^TJ$ a useful curvature approximation.

When it struggles

The method can propose steps that are too large when the starting point is poor. It also struggles when columns of $J$ are nearly dependent, which signals that parameters have similar effects on the curve.

Practical safeguards

Step limits, line searches, parameter scaling, and bounds can make the iteration safer. Levenberg–Marquardt adds a particularly important safeguard by damping directions that appear unreliable.

Derive the update from the local model

Minimize $\|\mathbf r+J\boldsymbol\delta\|^2$. Differentiating with respect to $\boldsymbol\delta$ gives

$2J^T(\mathbf r+J\boldsymbol\delta)=0,$

which yields the Gauss–Newton equations. This shows exactly which approximation is made: residuals are linearized, then that linear approximation is solved optimally.

Rank reveals learnability

If $J$ has nearly dependent columns, different parameter changes produce almost the same change in predictions. The numerical instability is therefore also a scientific warning: the data do not distinguish those parameter effects well.

Check your understanding

Why can Gauss–Newton be inaccurate when residuals remain large at the optimum?

Show the reasoning

The omitted Hessian terms are multiplied by residuals. When residuals are not small, those second-derivative contributions may be important, so $J^TJ$ is a poor curvature approximation.