Levenberg–Marquardt Balances Caution and Speed

Gauss–Newton can be wonderfully fast near a solution and unreliable far from one. The Levenberg–Marquardt method, usually abbreviated LM, modifies the step so the algorithm can be cautious when its local approximation is uncertain.

The damped system

A common form of the LM step is

$$(J^TJ+\lambda I)\boldsymbol\delta=-J^T\mathbf r,$$

where $\lambda\ge0$ is the damping parameter.

When $\lambda$ is small, the step resembles Gauss–Newton. When $\lambda$ is large, the step becomes shorter and points more like gradient descent.

Adapting the damping

After a trial step, the algorithm compares the actual reduction in squared error with the reduction predicted by the local model. A successful, well-predicted step usually allows $\lambda$ to decrease. A poor step is rejected or shortened by increasing $\lambda$.

This adaptive behavior is why LM is often a dependable default for small and medium unconstrained least-squares problems.

What LM does not solve

LM does not make parameters identifiable, guarantee the global minimum, or repair an inappropriate model. It also does not naturally enforce parameter bounds in its basic form; trust-region reflective methods are often preferred when hard bounds matter.

Scaling and stopping

Parameter scales affect the meaning of the damping term. Good implementations scale variables or use a diagonal matrix related to $J^TJ$. Stopping criteria commonly monitor the gradient, the parameter step, and the reduction in the objective.

See the damping in one dimension

For a scalar parameter, the step is

$\delta=-\frac{Jr}{J^2+\lambda}.$

If $J=-4$ and $r=5$, Gauss–Newton gives $1.25$. With $\lambda=16$, LM gives $20/32=0.625$. Damping has not changed the downhill direction; it has reduced trust in the local curvature model.

Damping is not regularization by default

The term $\lambda I$ resembles ridge regression, but LM changes $\lambda$ during optimization to control a step. Ridge keeps a penalty as part of the statistical objective. Confusing them leads to incorrect interpretations of parameter shrinkage.

Check your understanding

What happens to the LM step as $\lambda\to\infty$?

Show the reasoning

Its magnitude approaches zero and its direction approaches the negative gradient direction $-J^T\mathbf r$, scaled by $1/\lambda$. The algorithm becomes cautious gradient descent.