Chapter 07

PEM: Asymptotics, ARMAX & Identifiability

The deeper half of identification: what prediction-error minimisation converges to as data grows (consistency when the true system is in the model class, the best approximant otherwise), experimental vs structural identifiability, the uncertainty of the estimates, and the iterative maximum-likelihood algorithm that ARMAX's nonlinear predictor forces on us.

Reading: ~38 min Interactive: 1 widgets Source: Polimi MIDA1 2025/26 — Lecture 10 (Dynamic modelling, asymptotics & identifiability) · Polimi MIDA1 2025/26 — Lecture 10A (Identification example) · Polimi MIDA1 2025/26 — Lecture 10B (Uncertainty of the parameter estimates) · Polimi MIDA1 2025/26 — Lecture 12 (PEM identification of ARMAX models)

01 · Theory

Asymptotic analysis of PEM

The estimate $\hat\vartheta_N$ is itself random — it depends on the data. As data grows, where does it go? The answer decides whether identification recovers the truth or merely the best available approximation.

Because the residual is a stationary ergodic process, the empirical cost converges to the expected one: $J_N(\vartheta)\to\bar J(\vartheta)=\mathbb{E}[\varepsilon(t;\vartheta)^2]$ , and (if the convergence is regular) the minimiser $\hat\vartheta_N$ tends to the set $\Delta$ of minimisers of $\bar J$ . The central object is the innovation $e(t)=y(t)-\hat y(t;\vartheta^\circ)$ — the error the true optimal predictor still makes. It is independent of the past, which forces $\vartheta^\circ$ to be a minimiser of $\bar J$ .

"Explain the asymptotic properties of PEM"

Asked as an open-ended theory question repeatedly (January 2026, June 2026 Problem 4.1). Graders want the convergence $J_N\to\bar J$ , the innovation argument that $\vartheta^\circ\in\Delta$ , and the four cases below. Learn it as a structured derivation, not a slogan.

Derivation Why θ° minimises J̄, and the four cases

Split the residual of a generic model around the true one: $\varepsilon(t;\vartheta)=\underbrace{\big[y(t)-\hat y(t;\vartheta^\circ)\big]}_{e(t)\ \text{(innovation)}}+\big[\hat y(t;\vartheta^\circ)-\hat y(t;\vartheta)\big].$ The innovation $e(t)$ is uncorrelated with the past; the second bracket depends only on the past; so the two are uncorrelated and their variances add: $\bar J(\vartheta)=\mathrm{Var}[e(t)]+\mathrm{Var}\big[\hat y(t;\vartheta^\circ)-\hat y(t;\vartheta)\big]\ \ge\ \mathrm{Var}[e(t)]=\bar J(\vartheta^\circ).$ So $\vartheta^\circ\in\Delta$ . Four cases follow:

$S\in M$ , $\Delta$ a singleton → consistent: $\hat\vartheta_N\to\vartheta^\circ$ .
$S\in M$ , $\Delta$ not a singleton → converges to some equivalent parameterisation.
$S\notin M$ , $\Delta$ a singleton → converges to the unique best approximant in $M$ .
$S\notin M$ , $\Delta$ not a singleton → wanders within the set of best approximants.

In practice $S\in M$ is rare, so PEM delivers the best in-class approximant (minimum asymptotic prediction-error variance).

2026-01-q22026Asymptotic analysis of PEMhard6 pts

Explain and discuss the main results of the asymptotic analysis of PEM methods.

2024-01-q32024PEM consistency (S in M)medium5 pts

Prove qualitatively that when the model set contains the true system dynamics, the PEM estimate converges to the true system, and explain what happens otherwise.

02 · Core

Identifiability: the experiment and the model

A system is identifiable in a model family when its parameters can be estimated unambiguously ( $\Delta$ a singleton). Two things can spoil that, and they are examined separately:

Only the last one is silent. An over-parameterised family announces itself as a whole set of equivalent fits; an under-parameterised one returns a single confident estimate that is simply wrong, and the only tell is the coloured residual.

Experimental identifiability

Depends on the experiment $E$ . If the input does not excite a mode, no method recovers it — with $u\equiv0$ , $G(z)$ is completely unidentifiable. The input must be persistently exciting (a PRBS is the standard rich choice).

Structural identifiability

Depends on the model family $M$ . An over-parameterised family (fitting ARMAX(2,2,2) to an ARMAX(1,1,1) system) has infinitely many equivalent models however you design $E$ ; an under-parameterised one has a unique best approximant that is not the true system.

Noise is good — and the S₁/S₂ trap

Two counter-intuitive facts the exams love. (1) Noise helps identification: it excites the dynamics, so even a step input can give consistent ARX estimates when $\lambda^2>0$ . (2) The same ARX regression is consistent for an equation-error system $S_1$ but biased for an output-error system $S_2$ — the estimate $\hat a$ converges to a wrong value (e.g. $0.588$ ) no matter how much data you collect, because the coloured residual correlates with the regressor. Structural mismatch, not sample size, is the culprit.

The clean fingerprint of a good fit is, once more, a white residual. When you fit an AR/ARX model to a process that is really an MA (or ARMAX), the residual stays coloured — the tell that the true system is outside your class and you are seeing a best approximant, not the truth.

Exam-format worked example Best AR(1) approximant of an MA(1) — by PEM

Setup

Data come from an MA(1) $y(t)=e(t)+\tfrac12 e(t-1)$ , $e\sim\mathrm{WN}(0,1)$ ; fit the class $M:\ y(t)=a\,y(t-1)+\xi(t)$ . Since $S\notin M$ , expect a best approximant with a coloured residual.

Covariances of the data

$\gamma(0)=1+\tfrac14=\tfrac54$ , $\gamma(1)=\tfrac12$ (MA(1): only lags $0,1$ survive).

PEM estimate

The AR(1) predictor $\hat y=a\,y(t-1)$ is linear, so $J(a)=(1+a^2)\gamma(0)-2a\gamma(1)$ ; $J'(a)=0$ gives $\hat a=\dfrac{\gamma(1)}{\gamma(0)}=\dfrac{1/2}{5/4}=\dfrac25$ — the lag-1 correlation (Yule–Walker).

Residual variance

$\hat\lambda^2=J(\hat a)=\gamma(0)-\dfrac{\gamma(1)^2}{\gamma(0)}=\tfrac54-\dfrac{1/4}{5/4}=\dfrac{21}{20}$ — strictly above the true innovation variance $1$ , the price of $S\notin M$ .

2025-02-q32025Q03PEM identification & model comparisonhard8 pts

The true system is $S:\;y(t)=-\tfrac12 y(t-1)+e(t)$, $e\sim\mathrm{WN}(0,1)$. (a) Compute $\gamma(\tau)$ for $\tau=0,1,2$. (b) How would (a) change if $E[e(t)]=1$? (c) With model class $M_1:\;y(t)=a\,y(t-2)+\xi(t)$, find $\hat a$ by PEM. (d) With $M_2:\;y(t)=-\tfrac12 y(t-1)+\xi(t)+b\,\xi(t-1)$, find $\hat b$ by PEM. (e) Which model is best? (f) Interpret via the transfer functions.

2026-06-q32026Q03PEM identification (AR model of an MA process)medium7 pts

The data are generated by an MA(1) $S:\;y(t)=e(t)+\tfrac12 e(t-1)$, $e\sim\mathrm{WN}(0,1)$, and fitted with $M:\;y(t)=a\,y(t-1)+\xi(t)$. (a) Compute $\gamma(0),\gamma(1)$. (b) Find the PEM estimate $\hat a$. (c) Find $\hat\lambda^2$.

2026-07-q32026Q03PEM identification (AR model of an MA process)hard8 pts

The data are generated by $S:\;y(t)=e(t)+e(t-1)-\tfrac12 e(t-2)$, $e(\cdot)\sim\mathrm{WN}(0,1)$ (an MA(2)), and are fitted with the model class $M:\;y(t)=a\,y(t-1)+b\,y(t-2)+\xi(t)$, $\xi\sim\mathrm{WN}(0,\lambda^2)$. (a) Compute $\gamma(0),\gamma(1),\gamma(2)$ of the true process. (b) Write the PEM criterion $J(a,b)$. (c) Find the PEM estimates $\hat a,\hat b$. (d) Find the estimated noise variance $\hat\lambda^2$.

03 · Precision

How precise is the estimate?

Even when consistent, $\hat\vartheta_N$ has finite-sample scatter. Define the pseudo-regressor $\psi(t)=\partial\hat y(t)/\partial\vartheta$ (the gradient of the predictor). Then for large $N$ the estimate is asymptotically Gaussian around the truth:

Estimate covariance

$\sqrt{N}\,(\hat\vartheta_N-\vartheta^\circ) \xrightarrow{\ \text{as}\ } \mathcal{G}(0,\bar P), \qquad \bar P = \mathrm{Var}[\varepsilon(t;\vartheta^\circ)]\cdot\bar R^{-1}, \quad \bar R = \mathbb{E}[\psi\psi^\top].$

So the covariance is $\bar P/N$ : precision improves like $1/N$ , and worsens where the regressors carry little information ( $\bar R$ near-singular). For an AR(1), this specialises to the clean result $\mathrm{Var}[\hat a]\approx (1-a^2)/N$ — a pole near the unit circle is estimated more precisely. In practice $\bar R$ and $\mathrm{Var}[\varepsilon]$ are replaced by their sample averages over the data.

04 · Recipe

ARMAX by maximum likelihood

For ARMAX $A(z)y=B(z)u(t-k)+C(z)\xi(t)$ , the one-step predictor carries a $1/C(z)$ factor, so it is nonlinear in $\vartheta$ and $J_N$ is not quadratic — least squares is out. Instead minimise $J_N$ iteratively with Newton / Gauss–Newton (the maximum-likelihood algorithm):

Algorithm One Gauss–Newton (ML) step

Residual & gradient at the current θ

Compute the residual $\varepsilon(t)$ by running the data through the current predictor, and the pseudo-regressor $\psi(t)=-\partial\varepsilon(t)/\partial\vartheta$ — which turns out to be the signals $y$ , $u$ , $\varepsilon$ each filtered by $1/\hat C(z)$ .

Update

$\vartheta^{(i+1)}=\vartheta^{(i)}+\big(\sum_t\psi\psi^\top\big)^{-1}\sum_t\psi\,\varepsilon$ — a least-squares step on the linearised problem, structurally the ARX normal equations with $\psi$ in place of $\varphi$ .

Initialise and validate

Start from a high-order ARX (long-division) estimate — a good basin avoids local minima — iterate to convergence, then check the residual is white. The MA part is what made this necessary; the AR/ARX case needed no iteration at all.

"Explain the identification of an MA-with-exogenous (ARMAX) model"

A recurring open question (July 2025). A complete answer says why LS fails (nonlinear predictor), gives the Gauss–Newton update with the $1/C$ -filtered pseudo-regressor $\psi$ , and notes the ARX initialisation and the whiteness check.

2025-07-q32025Identifying ARMAX / MAX modelshard6 pts

Explain in detail the methodology for identifying a model whose noise enters through a moving-average (i.e. ARMAX / MAX), and why least squares does not suffice.

You fit an AR(1) to data that are actually an MA(1), and the residual comes out coloured, not white. What does this tell you?

Load-bearing ideas

Asymptotics: $J_N\to\bar J=\mathbb{E}[\varepsilon^2]$ , $\hat\vartheta_N\to\Delta$ ; the innovation argument puts $\vartheta^\circ\in\Delta$ . Four cases: consistent only when $S\in M$ with a unique minimum; otherwise the best approximant.
Identifiability = experimental (the input must persistently excite) + structural (the family must be neither over- nor under-parameterised). Noise helps; a coloured residual signals $S\notin M$ .
Uncertainty: $\sqrt N(\hat\vartheta_N-\vartheta^\circ)\to\mathcal G(0,\bar P)$ , covariance $\bar P/N=\mathrm{Var}[\varepsilon]\,\bar R^{-1}/N$ ; for AR(1), $\mathrm{Var}[\hat a]\approx(1-a^2)/N$ .
ARMAX is nonlinear in $\vartheta$ ⇒ iterative Gauss–Newton/ML with the $1/C$ -filtered pseudo-regressor $\psi$ , initialised from an ARX fit.

Exam radar. Prepare the asymptotic-PEM open answer (four cases + innovation argument), the best-approximant computation (fit AR to MA: $\hat a=\gamma(1)/\gamma(0)$ , coloured residual), and the ARMAX-ML methodology answer.