Goodness of Fit Requires More Than R²

No single number answers “Is this model good?” Some measures describe in-sample error, some compare complexity, and others estimate performance on new data. A responsible assessment uses several views that match the purpose of the model.

Error on the response scale

The residual standard error is

$$s=\sqrt{\frac{\mathrm{SSE}}{n-p}},$$

where $p$ is the number of estimated coefficients. The root mean squared error is often reported as

$$\mathrm{RMSE}=\sqrt{\frac{1}{n}\sum_i e_i^2}.$$

Both have the same units as the response, which makes them directly interpretable.

Explained variation

For ordinary least squares with an intercept,

$$R^2=1-\frac{\mathrm{SSE}}{\mathrm{SST}}.$$

It describes the fraction of sample variation accounted for relative to predicting every observation by $\bar y$.

Adjusted $R^2$ penalizes the automatic improvement produced by adding predictors:

$$R^2_{\mathrm{adj}}=1-\frac{\mathrm{SSE}/(n-p)}{\mathrm{SST}/(n-1)}.$$

Complexity-aware comparison

AIC and BIC combine likelihood-based fit with a penalty for parameter count. They are relative measures: their absolute values are not meaningful without other models fitted to the same response data under comparable assumptions.

Prediction on new data

A holdout set or cross-validation estimates performance beyond the training sample. Keep the full fitting procedure—including transformations and tuning—inside each training fold to avoid information leakage.

Residual diagnostics

Plot residuals against fitted values and important predictors. Look for curvature, changing spread, clusters, temporal dependence, and unusual observations. A normal quantile plot helps assess the error distribution when normal-based intervals or tests matter.

Match the diagnostic to the claim

If the claim is accurate prediction, prioritize held-out error and calibration. If it is parameter interpretation, prioritize identifiability, standard errors, and assumption checks. If it is mechanistic adequacy, inspect residual structure and whether parameters remain stable across conditions.

Build an evidence table

Record response-scale error, uncertainty coverage, residual pattern, sensitivity to influential data, out-of-sample error, and scientific plausibility. A model “passes” only relative to a specified use and tolerance.

Check your understanding

Can two models have similar cross-validated RMSE but different scientific value?

Show the reasoning

Yes. One may have interpretable, stable parameters and respect known constraints; another may predict equally well but violate mechanism or behave implausibly outside the sample.