Curve Fitting: From Data to Model

Curve fitting turns observations into a mathematical relationship. Given pairs $(x_i,y_i)$, we choose a function $f(x;\boldsymbol\theta)$ and estimate parameters $\boldsymbol\theta$ so that the function follows the data. The calculation matters, but the most important decisions occur before and after it.

1. Begin with the question

Decide what the model must do. Interpolation estimates values inside the observed range. Prediction estimates new outcomes. Scientific parameter estimation assigns physical meaning to coefficients. These goals may favor different models and different standards of evidence.

2. Inspect the data

Plot the observations. Check units, impossible values, repeated measurements, missing data, and the range of the predictor. A transformed axis may reveal a pattern hidden on the original scale.

3. Choose a model family

A straight line uses

$$y=\beta_0+\beta_1x+\varepsilon.$$

Other relationships may be polynomial, exponential, logistic, periodic, or derived from a physical law. Prefer the simplest family that can represent the structure the question requires.

4. Define the residuals

For a fitted model $\hat y_i=f(x_i;\hat{\boldsymbol\theta})$, the residual is

$$e_i=y_i-\hat y_i.$$

Least squares chooses parameters that minimize $\sum e_i^2$. If observations have unequal uncertainty, weighted least squares minimizes $\sum w_i e_i^2$ instead.

5. Fit, diagnose, and compare

After estimating parameters, inspect residual plots, uncertainty intervals, and sensitivity to unusual observations. Compare plausible models using out-of-sample performance or complexity-aware measures—not only the error on the data used for fitting.

6. Report the model honestly

State the equation, parameter estimates, units, uncertainty, data range, fitting method, and important assumptions. Avoid extrapolating far beyond the observations unless theory strongly supports the model there.

A decision-oriented workflow

Before fitting, write a one-sentence estimand: “Estimate the mean response at $x=5$,” “predict a new response,” or “estimate the physical decay constant.” Then identify the observation unit, likely dependence, measurement uncertainty, and scientifically permissible parameter range. These choices determine the objective and validation strategy.

Separate three kinds of uncertainty

Parameter uncertainty concerns unknown coefficients. Mean-response uncertainty concerns the estimated average curve. Prediction uncertainty additionally includes irreducible scatter for a new observation and is therefore wider. Reporting one as another overstates precision.

Check your understanding

Why should preprocessing decisions be repeated inside each cross-validation fold?

Show the reasoning

If scaling, transformation selection, or outlier rules use the full dataset, information from validation observations leaks into training. The resulting performance estimate is optimistically biased.