Chapter 06

Identification: Least Squares & ARX

The linear half of system identification: the least-squares estimator and its normal equations, the disturbance-based taxonomy of model families, the predictive approach that turns "which model is best" into an optimisation, and the key result that an ARX predictor is a linear regression you solve in one shot.

Reading: ~34 min Interactive: 1 widgets Source: Polimi MIDA1 2025/26 — Lecture 9 (Static modelling) · Polimi MIDA1 2025/26 — Lecture 10 (Dynamic modelling) · Polimi MIDA1 2025/26 — Lecture 11 (PEM identification of ARX models) · Polimi MIDA1 2025/26 — Appendix C (Least Squares)

01 · Base case

Least squares and the normal equations

Identification is an optimisation: pick the model in a chosen family that best predicts the data. When the predictor is linear in the parameters, that optimisation has a one-line closed-form answer — least squares.

A linear regression model writes the output as a known regressor vector times unknown parameters plus noise, $y(t)=\varphi(t)^\top\vartheta+v(t)$ . The least-squares estimate minimises the sum of squared prediction errors $J_N(\vartheta)=\frac1N\sum_t(y(t)-\varphi(t)^\top\vartheta)^2$ . Since $J_N$ is quadratic in $\vartheta$ , setting its gradient to zero gives the normal equations:

Normal equations

$R_N\,\hat\vartheta = \sum_t \varphi(t)\,y(t), \qquad R_N = \sum_t \varphi(t)\varphi(t)^\top, \qquad \hat\vartheta = R_N^{-1}\sum_t \varphi(t)\,y(t).$

"Describe the least-squares estimation procedure"

This derivation is a recurring open-ended exam question (e.g. June 2026, July 2026 Problem 4.1). A full answer states the model, the quadratic cost, the normal equations, the uniqueness condition (below), and the consistency property. Write it as a self-contained derivation, not a sentence.

✓

When the solution is unique

$\hat\vartheta$ is unique iff $R_N=\sum_t\varphi\varphi^\top$ is non-singular. A singular $R_N$ is not a numerical accident — it means the data are not informative enough to separate all parameters (no persistent excitation), and infinitely many $\vartheta$ give the same cost. Under persistent excitation and a residual uncorrelated with the regressors, $\hat\vartheta$ is unbiased and consistent, with a covariance that shrinks as $N$ grows.

Aside What goes wrong if the disturbance is not white

Least squares is unbiased when the residual $v(t)$ is uncorrelated with the regressor $\varphi(t)$ . For an ARX model the regressor is made of past data, so a white equation noise is uncorrelated with it and LS is consistent. But if the true disturbance is coloured (e.g. an output-error or ARMAX noise), the regressor and residual correlate, LS acquires a bias that never vanishes, and a different method is needed — the theme of the identifiability discussion and the next chapter.

2025-06-q22025Least squares estimationmedium5 pts

Explain the least-squares estimation procedure for a linear regression model, including the normal equations, when the solution is unique, and its statistical properties.

02 · Taxonomy

Model families: where the disturbance enters

Dynamic models are classified by how the noise enters — and that choice decides which estimation algorithm you can use.

One summing junction moves; everything else about the estimation problem changes with it.

Output error (OE)

$y(t)=G(z)u(t-k)+\eta(t)$ — noise added only at the output. All model error is measurement noise.

Equation error (AR / ARX)

$A(z)y(t)=B(z)u(t-k)+\eta(t)$ — white noise in the equation itself. The predictor is linear in the parameters.

ARMAX

$A(z)y(t)=B(z)u(t-k)+C(z)\eta(t)$ — an MA-shaped equation noise. Richer, but the predictor is nonlinear.

ARXAR / ARIMAX / FIR

AR-shaped residual ( $1/D(z)$ ); a random-walk drift (ARIMA/CARIMA); or $n_a=0$ , the finite impulse response $y=B(z)u(t-1)+\eta$ .

Choosing a family balances purpose, flexibility, parsimony, and algorithm complexity — a simpler model gives more robust estimates, a richer one fits more dynamics but may have local minima or be unidentifiable.

03 · Core

The predictive approach

How do you compare a stochastic model with numeric data? You cannot compare a random variable to a number — but you can compare its prediction to the datum. That is the predictive approach: a model is good if its one-step-ahead predictions are accurate.

Prediction error & cost

$\varepsilon(t;\vartheta) = y(t) - \hat y(t\mid t-1;\vartheta), \qquad J_N(\vartheta) = \frac1N\sum_{t} \varepsilon(t;\vartheta)^2.$

The optimal model minimises $J_N$ over the admissible parameters. But $J_N$ is only a partial score: you must also check that the residual is white (Chapter 8) — otherwise predictable dynamics remain and the model is incomplete. Notice this is the same “make the residual white” criterion from Chapter 1, now driving identification.

2026-02-q22026Choosing an identification methodmedium6 pts

Explain how you would approach the identification of a dynamical model — what determines whether you can use least squares or need an iterative method, and what the workflow is.

04 · Key result

ARX is a linear regression → least squares

Here is the result that makes ARX the workhorse. Its one-step predictor is a linear combination of past data — no dependence on past predictions — so it is a linear regression and LS solves it in one shot:

ARX prediction form

$\hat y(t\mid t-1) = \varphi(t)^\top\vartheta, \quad \varphi(t) = \big[\,y(t-1)\dots y(t-n_a)\ \ u(t-k)\dots u(t-k-n_b)\,\big]^\top, \quad \vartheta = \big[\,a_1\dots a_{n_a}\ b_0\dots b_{n_b}\,\big]^\top.$

The predictor is always stable (it is not recursive in its own output). Contrast the neighbours: an ARXAR predictor is bilinear — fix $D(z)$ and it is linear in $(a_i,b_i)$ , fix $A,B$ and it is linear in $d_i$ — so you alternate least squares. An ARMAX predictor is genuinely nonlinear in the parameters, and needs the iterative method of the next chapter.

Exam-format worked example Identify an ARX parameter by least squares from data

The model and data

$M:\ y(t)=V\,y(t-1)+u(t-1)+e(t)$ with the single unknown $V$ , and outputs $y(1..4)=0.1,-0.4,2,-1$ . Stationarity needs the pole inside the unit circle, so $|V|<1$ .

The regression

The predictor $\hat y(t\mid t-1)=V\,y(t-1)+u(t-1)$ is linear in $V$ ; with $u=0$ the regressor is $\varphi(t)=y(t-1)$ . So $\hat V$ is the scalar least-squares ratio.

Compute the ratio

$\hat V=\dfrac{\sum_t y(t)y(t-1)}{\sum_t y(t-1)^2}$ over $t=2,3,4$ . Numerator $=(-0.4)(0.1)+(2)(-0.4)+(-1)(2)=-2.84$ ; denominator $=0.1^2+0.4^2+2^2=4.17$ ; so $\hat V\approx-0.68$ .

A known input

With a measured exogenous $u(t)$ , subtract the known $u(t-1)$ term from $y(t)$ before regressing — the predictor stays linear in $V$ , so one-shot LS still applies. A richer $u$ only improves the conditioning of $R_N$ .

2024-01-q22024Q02ARX identification by least squaresmedium8 pts

Consider the model class $M:\;y(t)=V\,y(t-1)+u(t-1)+e(t)$, $e\sim\mathrm{WN}(0,\lambda^2)$, $V\in\mathbb{R}$, with measured outputs $y(1)=0.1,\ y(2)=-0.4,\ y(3)=2,\ y(4)=-1$. (a) For which $V$ is the process stationary (for stationary $u$)? (b) Write the time-domain one-step-ahead predictor. (c) Identify $V$ by least squares with $u(t)=0$. (d) Would the answer change if $u(t)=\sin(3t)$?

You want to identify an ARMAX(1,1,1) model. Can you use one-shot least squares?

Load-bearing ideas

Least squares minimises the quadratic prediction-error cost; the normal equations $R_N\hat\vartheta=\sum\varphi y$ give $\hat\vartheta=R_N^{-1}\sum\varphi y$ , unique iff $R_N$ is non-singular (persistent excitation).
Model families are classified by where the disturbance enters — OE, ARX, ARMAX, ARXAR, FIR — and that choice sets the algorithm.
The predictive approach scores a model by its one-step prediction error $J_N$ , then checks residual whiteness.
ARX is a linear regression ⇒ one-shot LS, always-stable predictor. ARXAR ⇒ alternating LS; ARMAX ⇒ nonlinear, iterative (next chapter).

Exam radar. Be ready to (i) write the LS derivation as an open answer, (ii) set up and solve an ARX least-squares estimate from a short data table, and (iii) say why you can use LS for ARX but not for ARMAX. The linear/nonlinear split is the organising idea of identification.