AIC and BIC Balance Fit Against Complexity

Better in-sample fit can always be purchased with more flexibility. Information criteria compare this improvement with a penalty for model size.

$\mathrm{AIC}=2k-2\log L,\qquad \mathrm{BIC}=k\log n-2\log L,$

where $k$ is the number of estimated parameters, $n$ is sample size, and $L$ is the maximized likelihood.

Relative, not absolute

Smaller values are preferred, but only differences among models fitted to the same response data are meaningful. AIC and BIC do not have universal “good” thresholds.

Different penalties

BIC’s penalty grows with sample size and often favors smaller models more strongly. AIC is motivated by expected predictive information loss; BIC is connected to selecting among candidate models under stronger assumptions.

Work with differences and weights

Suppose three models have AIC values $120$, $122$, and $130$. Subtract the minimum to obtain $\Delta=(0,2,10)$. Relative support is proportional to $e^{-\Delta/2}$, giving approximately $1$, $0.368$, and $0.0067$. Normalizing produces Akaike weights of about $0.727$, $0.268$, and $0.005$.

Assumptions travel with the likelihood

For Gaussian regression, the likelihood depends on the error model and whether variance is estimated. AIC values from least squares are comparable only when they arise from consistent likelihood definitions. Small samples may require AICc, whose extra correction is substantial when $n$ is not large relative to $k$.

Check your understanding

If every candidate model is badly misspecified, does the smallest AIC identify a good model?

Show the reasoning

No. It identifies the best-supported member of the candidate set under the criterion. Absolute adequacy still requires diagnostics, subject knowledge, and validation.