Chapter 05

The Prediction Problem

The Kolmogorov–Wiener theory that is the heart of MIDA1: the optimal k-step predictor of an ARMA/ARMAX process, built by long division of C/A (equivalently the Diophantine equation), read from data through the whitening filter, with its error variance — and why the canonical form is mandatory.

Reading: ~38 min Interactive: 1 widgets Source: Polimi MIDA1 2025/26 — Lecture 8 (The prediction problem)

01 · Setup

Optimal prediction from the noise

Write a process as white noise shaped by a filter, and predicting it becomes a bookkeeping problem: separate what the future noise will bring (unknowable) from what the past has already fixed (your predictor).

Take a canonical process $v(t)=W(z)\varepsilon(t)$ with $W(z)=w_0+w_1z^{-1}+w_2z^{-2}+\dots$ , so $v(t)=\sum_{i\ge0}w_i\varepsilon(t-i)$ . To predict $r$ steps ahead, split the sum at the present:

Split

$v(t+r) = \underbrace{w_0\varepsilon(t+r)+\dots+w_{r-1}\varepsilon(t+1)}_{\text{future noise — unpredictable}} + \underbrace{w_r\varepsilon(t)+w_{r+1}\varepsilon(t-1)+\dots}_{\text{depends on the past}}.$

The first group is future white noise, uncorrelated with everything up to time $t$ , so its best estimate is its mean, zero. The second group is known once the past is known. Hence the optimal predictor and its error are

Optimal predictor

$\hat v(t+r\mid t) = w_r\varepsilon(t)+w_{r+1}\varepsilon(t-1)+\dots, \qquad \mathrm{Var}[\varepsilon(t+r)] = (w_0^2+w_1^2+\dots+w_{r-1}^2)\,\lambda^2.$

The error variance grows with the horizon $r$ (more unknown noise terms accumulate) and tends to the process variance as $r\to\infty$ — far ahead, the best you can do is the mean. The predictor filter is the tail of $W$ : $\hat W_r(z)=w_r+w_{r+1}z^{-1}+\dots = F_r(z)/A(z)$ , obtained by dividing $C$ by $A$ for $r$ steps.

02 · Recipe

Long division: the exam’s core mechanic

Almost every prediction problem is: canonicalise, long-divide, read off the predictor and the error variance. Here it is end to end on a clean ARMA(1,1).

The division is doing one job: separating the noise you have already seen from the noise you have not. Keep the picture in view while the algebra runs — it is what every line of it means.

Exam-format worked example k-step prediction by long division

The process

$y(t)=0.5\,y(t-1)+e(t)+0.2\,e(t-1)$ , $e\sim\mathrm{WN}(0,1)$ , i.e. $A(z)=1-0.5z^{-1}$ , $C(z)=1+0.2z^{-1}$ . Pole $0.5$ and zero $-0.2$ are both inside the unit circle, so it is canonical — always check this first.

Expand W(z)=C/A as a power series

Match coefficients in $C=A\cdot W$ with $W=1+w_1z^{-1}+w_2z^{-2}+\dots$ : the $z^{-1}$ term gives $0.2=w_1-0.5\Rightarrow w_1=0.7$ ; the $z^{-2}$ term gives $0=w_2-0.5\,w_1\Rightarrow w_2=0.35$ . So $W(z)=1+0.7z^{-1}+0.35z^{-2}+\dots$

Two-step predictor

Keep the tail from $w_2$ : $\hat W_2(z)=w_2+w_3z^{-1}+\dots$ , and the predictor from data is $\hat y(t+2\mid t)=\dfrac{F_2(z)}{C(z)}y(t)$ where $F_2$ is the remainder of the two-step division.

Error variance

The two-step error keeps the first two innovations: $\mathrm{Var}[\varepsilon(t+2)]=(w_0^2+w_1^2)\lambda^2 =(1+0.49)\cdot1=1.49$ . (One step ahead it would be $w_0^2\lambda^2=1$ — the innovation itself.)

Problem 2 on every paper

Prediction is the second problem of every structured MIDA1 exam and appears across the archive back to 2022 — show the process is canonical, perform the long division for two steps, give the optimal 1- and 2-step predictors, and compute the error variance, all as boxed numbers. The two worked exam problems below are exactly this. It is the single most reliable source of marks in the course.

03 · Core

From data: the whitening filter

The predictor above uses the noise $\varepsilon$ , which we never observe — we have the signal $v$ . The bridge is the whitening filter: since $v=W(z)\varepsilon$ with $W=C/A$ canonical,

Whitening

$\varepsilon(t) = W(z)^{-1}v(t) = \frac{A(z)}{C(z)}\,v(t).$

Cascading the whitening filter with the predictor-from-noise gives the predictor from data, $\hat v(t+r\mid t)=\dfrac{F_r(z)}{C(z)}\,v(t)$ — same numerator $F_r$ as the noise predictor, but denominator $C$ instead of $A$ . This is exactly why $C(z)$ must have its roots inside the unit circle (canonical form): the predictor is a filter with denominator $C$ , and it must be stable, i.e. the predictor’s stability depends only on $C(z)$ .

Derivation The Diophantine equation — why long division works

Long division of $C$ by $A$ for $r$ steps is the same as solving the Diophantine equation

$C(z) = A(z)\,E(z) + z^{-r}F_r(z),\qquad E(z)=e_0+e_1z^{-1}+\dots+e_{r-1}z^{-(r-1)},\ e_0=1.$

Multiply the ARMA model $A\,v=C\,\varepsilon$ through by $E$ and substitute $A E = C - z^{-r}F_r$ :

$C(z)\,v(t+r) = C(z)E(z)\,\varepsilon(t+r) + F_r(z)\,v(t).$

On the right, $C(z)E(z)\varepsilon(t+r)=e_0\varepsilon(t+r)+\dots+e_{r-1}\varepsilon(t+1)$ is future noise — unpredictable — while $F_r(z)v(t)$ depends only on the past. Taking the second term as the predictor leaves exactly that future-noise residual, so it is optimal:

$\hat v(t+r\mid t) = \frac{F_r(z)}{C(z)}\,v(t), \qquad \mathrm{Var}[\varepsilon] = (e_0^2+\dots+e_{r-1}^2)\lambda^2.$

2026-07-q22026Q02Optimal prediction (long division)hard8 pts

Consider $y(t)=\tfrac12 y(t-1)-\tfrac18 y(t-2)+e(t)+\tfrac14 e(t-1)$, $e(\cdot)\sim\mathrm{WN}(0,1)$, i.e. $A(z)y(t)=C(z)e(t)$ with $A(z)=1-\tfrac12 z^{-1}+\tfrac18 z^{-2}$, $C(z)=1+\tfrac14 z^{-1}$. (a) Show the process is in canonical form. (b) Perform the long division of $C/A$ for two steps, writing $\tfrac{C(z)}{A(z)}=1+\alpha z^{-1}+z^{-2}\tfrac{\beta+\gamma z^{-1}}{A(z)}$. (c) Give the optimal two-step-ahead predictor. (d) Compute the two-step prediction-error variance.

2024-01-q12024Q01Prediction via canonical form (all-pass whitening)hard8 pts

A process $y(t)$ is generated by the scheme: white noise $e(\cdot)\sim\mathrm{WN}(0,1)$ is filtered by $\dfrac{1+5z^{-1}}{1+\tfrac15 z^{-1}}$ to give $\eta(t)$; then $\delta(t)=\eta(t)+w(t)$ with $w(\cdot)\sim\mathrm{WN}(0,1)$ and $e\perp w$; finally $y(t)=\dfrac{1}{1+\tfrac12 z^{-1}}\,\delta(t)$. (a) Is $y(t)$ stationary? (b) Write $y(t)$ in canonical form. (c) Compute the optimal two-step-ahead predictor $\hat y(t+2\mid t)$. (d) Compute the variance of the two-step prediction error.

04 · Pitfall & benchmark

Canonical form is mandatory; beat the stupid predictor

A non-canonical model gives a non-stationary predictor

If $C(z)$ has a root outside the unit circle and you plug it into the predictor formula, the denominator $C$ makes the predictor unstable — you would be tracking a stationary process with a non-stationary one. Fix it first: reflect the offending zero to its reciprocal and rescale $\lambda^2$ so $\gamma$ is unchanged (spectral factorization). Only then predict. Forgetting to canonicalise is the most common way to lose all of Problem 2.

How much does optimal prediction actually buy you? Compare against the stupid predictor $\hat v(t+1\mid t)=v(t)$ — “tomorrow equals today”. For an AR(1) with parameter $a$ , its mean-square error is $\tfrac{2}{1+a}\lambda^2$ , strictly worse than the optimal $\lambda^2$ whenever $a<1$ . The gap is the value of actually modelling the dynamics.

canonical-1step-predictorCanonical form & one-step predictormedium6 pts

Consider the ARMA(1,1) process $y(t)=\tfrac12 y(t-1)+e(t)+\tfrac14 e(t-1)$, $e(\cdot)\sim\mathrm{WN}(0,1)$, i.e. $A(z)=1-\tfrac12 z^{-1}$, $C(z)=1+\tfrac14 z^{-1}$. (a) Show the process is in canonical form. (b) Derive the optimal one-step-ahead predictor $\hat y(t\mid t-1)$. (c) Give the variance of the one-step prediction error.

2026-07-q52026Two-step prediction-error variancemedium

A canonical process gives the two-step-ahead predictor filter with quotient $1+\tfrac34 z^{-1}$ (so $w_0=1,\ w_1=\tfrac34$) and $e(\cdot)\sim\mathrm{WN}(0,1)$. Compute the variance of the two-step-ahead prediction error. Give the decimal value.

Your answer

05 · Formulas

Ready-to-use ARMA and ARMAX predictors

For a canonical ARMA process the one-step predictor has a compact closed form (no long division needed — it is the $r=1$ case):

1-step ARMA

$C(z)\,\hat v(t+1\mid t) = \big(C(z)-A(z)\big)\,v(t+1).$

Because both $A$ and $C$ are monic, $C-A$ has no constant term, so the right-hand side depends only on $v$ up to time $t$ — as a predictor must. The ARMAX $k$ -step predictor is the same idea with the input path added (Diophantine $C=AE+z^{-k}F_k$ ):

k-step ARMAX

$C(z)\,\hat y(t+k\mid t) = F_k(z)\,y(t) + B(z)E(z)\,u(t).$

The first term is the ARMA predictor; the second feeds in the measured input. Everything reduces to one long division of $C$ by $A$ .

A process has C(z)=1+1.25 z⁻¹ (zero at −1.25, outside the unit circle). You want the one-step predictor. What must you do first?

Load-bearing ideas

From noise: $\hat v(t+r\mid t)$ is the tail of $W=C/A$ from $w_r$ on; the error variance is $(w_0^2+\dots+w_{r-1}^2)\lambda^2$ , growing with the horizon toward $\gamma(0)$ .
Long division of $C$ by $A$ for $r$ steps (= the Diophantine equation $C=AE+z^{-r}F_r$ ) gives the predictor $F_r/C$ from data and the error variance from $E$ ‘s coefficients.
The whitening filter $A/C$ recovers the noise from the signal; the predictor’s stability depends only on $C(z)$ , so the process must be in canonical form first.
The stupid predictor $v(t)$ is the benchmark; optimal prediction beats it strictly.
ARMA one-step: $C\hat v(t+1\mid t)=(C-A)v(t+1)$ . ARMAX $k$ -step: add $B E\,u(t)$ .

Exam radar. This is the highest-yield chapter. Drill the StepThrough until canonicalise → long divide → predictor → error variance is automatic, and keep the “reflect an out-of-circle zero and rescale $\lambda^2$ ” move at your fingertips.