Nonlinear Curve Fitting Is an Iterative Search

A model can be curved in $x$ and still be linear in its parameters. True nonlinear least squares begins when the unknown parameters enter nonlinearly, as in

$$f(x;A,k)=Ae^{-kx}.$$

There is usually no single formula for the best estimates. The computer must search for them.

Residuals as a function of parameters

Write

$$r_i(\boldsymbol\theta)=y_i-f(x_i;\boldsymbol\theta), \qquad S(\boldsymbol\theta)=\sum_i r_i(\boldsymbol\theta)^2.$$

The goal is to find a low point on the objective surface $S$. Unlike linear least squares, this surface may contain long valleys, flat regions, or several local minima.

Starting values matter

An iterative method begins at a proposed parameter vector $\boldsymbol\theta_0$. Good starting values may come from a plot, a simplified model, physical bounds, or several deliberately different initial guesses.

A typical fitting loop

1. Evaluate predictions and residuals.
2. Calculate how predictions change with each parameter.
3. Propose a parameter step.
4. Accept, shorten, or modify the step.
5. Stop when changes in parameters, error, or gradient are sufficiently small.

Scaling is important. If one parameter is near $10^{-6}$ and another near $10^6$, the numerical problem may be poorly balanced even when the scientific model is reasonable.

Identifiability

Parameters are identifiable when the available data can distinguish their effects. If several parameter combinations produce nearly the same curve, individual estimates can be highly uncertain even when predictions look good.

After convergence

Plot the fit and residuals, compare runs from different starting values, inspect parameter correlations, and report uncertainty. Bounds can enforce physical constraints, but a parameter pressed against a bound is itself a diagnostic.

Learn from a profile of the objective

For $y_i=Ae^{-kt_i}+\varepsilon_i$, fix $k$ temporarily. The model is then linear in $A$, so the best $A$ can be found by linear least squares. Repeating across candidate $k$ values creates a profile $S(k)$. A sharp minimum means $k$ is well determined; a broad valley means many decay rates fit similarly.

A convergence checklist

Require agreement across multiple starts, a small gradient, stable predictions under small perturbations, plausible parameters, and residuals without structure. Report whether bounds were active and whether uncertainty came from local linearization, a profile, bootstrap, or another method.

Check your understanding

Why can a log transformation of $y=Ae^{-kx}$ change the fitted answer?

Show the reasoning

Fitting $\log y=\log A-kx$ minimizes squared errors on the log scale, implying multiplicative rather than additive errors. It also fails for nonpositive observations. The transformed and original objectives answer different statistical questions.