Chapter 08

Model Validation & Selection

Closing the identification loop: is the model good, and how complex should it be? The whiteness (Anderson) test on the residual with its confidence band, the cross-correlation test for the input path, and the FPE / AIC / MDL criteria and cross-validation for choosing the model order.

Reading: ~26 min Interactive: 1 widgets Source: Polimi MIDA1 2025/26 — Lecture 13 (Model validation and model selection)

01 · Core

The whiteness (Anderson) test

The whole course reduces to one check: is the residual white? A good model leaves a prediction error with no structure left in it. The Anderson test makes “no structure” quantitative.

The band drawn there is $\pmeta/\sqrt{N}=\pm0.098$ at $N=400$ . It narrows as data accumulates — more data, a stricter test.

Take the residual $\varepsilon(t)=y(t)-\hat y(t\mid t-1)$ of the identified model and form its sample normalised autocorrelation $\hat\rho(\tau)=\hat\gamma_\varepsilon(\tau)/\hat\gamma_\varepsilon(0)$ . Under the null hypothesis that $\varepsilon$ is white, for large $N$ each nonzero-lag value is approximately Gaussian:

Anderson

$\sqrt{N}\,\hat\rho(\tau) \approx \mathcal{N}(0,1) \quad\Rightarrow\quad \hat\rho(\tau) \text{ should lie within } \pm\frac{\beta}{\sqrt{N}}, \quad \beta\approx 1.96 \text{ (95\%)}.$

Procedure Running the Anderson test

Compute the residual autocorrelations

From the residual sequence, estimate $\hat\rho(\tau)$ for $\tau=1,2,\dots,m$ (a few dozen lags).

Draw the band

Plot the band $\pm\beta/\sqrt N$ around zero — narrower with more data, since it scales as $1/\sqrt N$ .

Read the verdict

If (almost) all $\hat\rho(\tau)$ fall inside the band, the residual is compatible with white noise — the model is validated. If several exceed it, the residual is still correlated: the order is too low, the structure inadequate, or dynamics were missed.

"Illustrate the whiteness / Anderson test"

This is one of the most frequent open-ended questions (July 2026 Problem 4.2, June 2025, and more). A full answer: why the residual must be white, the statistic $\hat\rho(\tau)$ , the distribution $\sqrt N\,\hat\rho(\tau)\sim\mathcal N(0,1)$ , the band $\pm\beta/\sqrt N$ , and the decision rule. Write it as a self-contained procedure.

2026-07-q42026Whiteness (Anderson) testmedium5 pts

Explain the whiteness test (Anderson test) for model validation — what it checks, the statistic used, and the confidence band.

A companion is the cross-correlation test: the residual should also be uncorrelated with past inputs, $\hat\rho_{\varepsilon u}(\tau)\approx 0$ . If it is not, the input-to-output path ( $B/A$ ) is mis-modelled even if the noise path looks fine.

02 · Selection

Choosing the model order

A bigger model always fits the training data better — the residual variance $\hat\lambda^2$ decreases monotonically with the number of parameters $p$ . So you cannot pick the order by $\hat\lambda^2$ alone; you need a criterion that penalises complexity.

FPE — Final Prediction Error

$\mathrm{FPE}=\hat\lambda^2\,\dfrac{N+p}{N-p}$ — inflates the residual variance by an over-fit factor. Minimise over $p$ .

AIC — Akaike

$\mathrm{AIC}=\ln\hat\lambda^2+\dfrac{2p}{N}$ — a fit term plus a $2p/N$ penalty.

MDL — Rissanen

$\mathrm{MDL}=\ln\hat\lambda^2+\dfrac{\ln N}{N}\,p$ — a heavier penalty ( $\ln N>2$ for $N>7$ ).

AIC over-selects; MDL is consistent

Because $\ln N>2$ for any realistic $N$ , MDL penalises complexity more than AIC, so it picks smaller orders — and it is consistent (it selects the true order as $N\to\infty$ ), whereas AIC tends to over-parameterise. If two criteria disagree, that difference is the reason. A common exam ask is to state all three and compare AIC vs MDL.

2024-01-q42024Model-order selection criteriamedium5 pts

When data are scarce and cross-validation is impractical, discuss at least two criteria for choosing the model order.

03 · Alternative

Cross-validation

The criteria above are cheap but assume the noise model. Cross-validation instead measures generalisation directly: split the data into a training set (fit by prediction-error minimisation) and a validation set (held out), and score the model by its prediction error on the unseen validation data. This simulates real forecasting — learn from the past, predict the future.

✓

Which tool when

Cross-validation — the most direct generalisation estimate, but needs enough data for both splits and is costlier.
Whiteness test — a diagnostic that the dynamics were captured, but not a direct accuracy measure.
FPE / AIC / MDL — closed-form, no held-out set; ideal when data are too scarce to split.

In practice they are combined: a criterion proposes an order, the whiteness test confirms the residual, and cross-validation checks it generalises.

2025-02-q22025Cross-validationmedium5 pts

In ARMA modelling, describe the cross-validation approach to model selection and compare it with alternatives (whiteness test, information criteria).

AIC and MDL disagree on the order of your model — AIC picks 5, MDL picks 3. Which is more likely to have over-parameterised, and why?

Load-bearing ideas

Whiteness (Anderson) test: the residual autocorrelation $\hat\rho(\tau)$ must lie inside the band $\pm\beta/\sqrt N$ ; exceedances mean leftover dynamics. The cross-correlation test checks the residual against past inputs.
Order selection: $\hat\lambda^2$ falls monotonically with $p$ , so penalise complexity — FPE $\hat\lambda^2\frac{N+p}{N-p}$ , AIC $\ln\hat\lambda^2+\frac{2p}{N}$ , MDL $\ln\hat\lambda^2+\frac{\ln N}{N}p$ . MDL penalises more, is consistent; AIC over-selects.
Cross-validation measures generalisation directly on held-out data; combine all three in practice.

Exam radar. The Anderson-test open answer (statistic + band + rule) and the FPE/AIC/MDL comparison are near-guaranteed. Keep the $\pm\beta/\sqrt N$ band and the AIC-over-selects / MDL-consistent contrast ready.