Chapter 03

MA, AR & ARMA Models

The model zoo that turns Wold's "white noise through a filter" into a handful of parameters: moving-average (MA), auto-regressive (AR) and ARMA processes, the monic normalisation, the Yule–Walker equations, and the covariance fingerprints that tell the three apart.

Reading: ~30 min Interactive: 1 widgets Source: Polimi MIDA1 2025/26 — Lecture 4 (MA, AR and ARMA processes)

01 · Model

The moving-average (MA) process

Filter white noise through a finite window and you get a moving-average process — the simplest way to build a signal whose samples are correlated over a short range.

An MA process of order $n$ is a weighted sum of the current and past $n$ white-noise samples:

MA(n)

$v(t) = c_0\,\eta(t) + c_1\,\eta(t-1) + \dots + c_n\,\eta(t-n),\qquad \eta(\cdot)\sim\mathrm{WN}(0,\lambda^2).$

Its mean is zero and — because white-noise samples at different times are uncorrelated — its variance and covariance are just sums of squared and shifted coefficients:

$\mathrm{Var}[v(t)] = (c_0^2 + c_1^2 + \dots + c_n^2)\,\lambda^2, \qquad \gamma(\tau) = 0 \ \text{ for } |\tau| > n.$

That cut-off is the MA fingerprint: the covariance is exactly zero beyond lag $n$ . In operatorial form $v(t)=C(z)\eta(t)$ with $C(z)=c_0+c_1z^{-1}+\dots+c_nz^{-n}$ — a filter with $n$ zeros and all its poles at the origin, so an MA process is always stationary. The pair $(c_i,\lambda^2)$ is redundant (scale the $c_i$ up and $\lambda$ down), so we fix $c_0=1$ : the polynomial $C(z)$ is monic.

✓

MA(∞) can model anything

Letting $n\to\infty$ , an MA( $\infty$ ) $v(t)=\sum_{i\ge0}c_i\eta(t-i)$ has finite variance $\sum_i c_i^2\lambda^2$ iff $\sum_i c_i^2 < \infty$ . Under that condition it is stationary — and its covariance can be arbitrarily long, so an MA( $\infty$ ) can represent any stationary process. That is exactly Wold’s non-deterministic part; AR and ARMA are finite-parameter shortcuts to the same place.

ma2-varianceMA(2) varianceeasy

For the MA(2) process $y(t)=e(t)+e(t-1)-\tfrac12 e(t-2)$ with $e(\cdot)\sim\mathrm{WN}(0,1)$, compute the variance $\gamma(0)$. Give the decimal value.

Your answer

02 · Model

The auto-regressive (AR) process & Yule–Walker

To get a covariance that is non-zero at every lag with only finitely many parameters, regress the process on its own past. An AR process of order $n$ is

AR(n)

$v(t) = a_1 v(t-1) + \dots + a_n v(t-n) + \eta(t), \qquad A(z)\,v(t) = \eta(t),\quad W(z) = \frac{1}{A(z)}.$

If $A(z)$ is stable (roots inside the unit circle) the AR process is a stationary MA( $\infty$ ). The relation between its parameters and its covariances is the single most-used tool in the course.

Exam-format worked example The Yule–Walker equations for an AR(1)

The trick

For $v(t)=a\,v(t-1)+\eta(t)$ , multiply both sides by $v(t-\tau)$ and take expectations. The key fact: $\mathbb{E}[\eta(t)v(t-\tau)]=\lambda^2$ if $\tau=0$ (the noise hits the current sample) and $0$ for $\tau>0$ (future noise is uncorrelated with the past).

τ = 0 and τ = 1

$\tau=0:\ \gamma(0) = a\,\gamma(1) + \lambda^2.$ $\qquad \tau=1:\ \gamma(1) = a\,\gamma(0).$

Solve

Substitute $\gamma(1)=a\gamma(0)$ into the first: $\gamma(0)=a^2\gamma(0)+\lambda^2$ , giving $\gamma(0) = \frac{\lambda^2}{1-a^2}, \qquad \gamma(\tau) = a^{\tau}\gamma(0).$ The covariance decays geometrically with the pole $a$ — never reaching zero. Smooth realizations for $a>0$ ; sign-flipping for $a<0$ .

Both directions

Read left-to-right and Yule–Walker gives covariances from parameters; read right-to-left (measure $\gamma(0),\gamma(1)$ , solve for $a,\lambda^2$ ) and it identifies the model — the route used in the time-series chapter. For AR( $n$ ) the same trick gives an $n\times n$ Toeplitz system.

ar1-variance-yule-walkerAR(1) variance (Yule-Walker)easy

For the AR(1) process $x(t)=\tfrac12 x(t-1)+e(t)$ with $e(\cdot)\sim\mathrm{WN}(0,1)$, compute the variance $\gamma_x(0)$. Give the decimal value.

Your answer

2025-09-q22025Yule-Walker equationseasy4 pts

Write the Yule-Walker equations for an AR(1) process and explain what they represent.

03 · Model

ARMA: poles and zeros together

An ARMA process combines both — an AR denominator and an MA numerator:

ARMA(na,nc)

$A(z)\,v(t) = C(z)\,\eta(t), \qquad W(z) = \frac{C(z)}{A(z)}.$

With a stable $A(z)$ it is stationary and, expanded, is again an MA( $\infty$ ). Its covariance eventually satisfies the AR recurrence — the vanishing covariance property: for lags beyond the MA order, $A(z)\gamma(\tau)=0$ , so $\gamma(\tau)\to 0$ as $\tau\to\infty$ . This gives the three covariance fingerprints that let you name a process by eye:

MA(n): γ cuts off

$\gamma(\tau)=0$ exactly for $|\tau|>n$ . A sharp edge in the covariance.

AR(n): γ tails off

$\gamma(\tau)$ decays geometrically, never exactly zero. (The dual fingerprint — the PARCOR cutting off — pins the AR order; see Chapter 9.)

ARMA: γ tails off

Like AR for large lags, after an initial MA-shaped transient of length $n_c$ .

2025-07-q12025Identifying process type from covarianceeasy1 pt

A zero-mean stationary process has autocovariance $\gamma(\tau)\ne 0$ only for $\tau=0$ and $\tau=\pm1$, and $\gamma(\tau)=0$ for $|\tau|\ge 2$. Which model class does it belong to?

04 · Extension

Adding an exogenous input: ARMAX

Time series describe a signal from its own past. Control and engineering systems also have an input we can measure — an exogenous variable. The ARMAX model adds it (renaming the output $y$ ):

Give the two paths separate denominators and it is no longer ARMAX — that is output error, with a different predictor and a different algorithm.

ARMAX

$A(z)\,y(t) = B(z)\,u(t-k) + C(z)\,\eta(t),$

with input polynomial $B(z)=b_0+b_1z^{-1}+\dots+b_{n_b}z^{-n_b}$ and input–output delay $k$ . The output is driven along two paths — a deterministic one through $B/A$ from the input, and a stochastic one through $C/A$ from the noise. Dropping $B$ recovers ARMA; dropping $C$ gives ARX (the workhorse of Chapter 6); and replacing the linear combination by a nonlinear $f(\cdot)$ gives NARMAX. This two-path picture is the object every identification method in Chapters 6–7 fits to data.

You measure a zero-mean signal and find γ(0)=5, γ(1)=2, γ(2)=1.3, γ(3)=0.85, decaying but never zero. Which model class fits best?

Load-bearing ideas

MA( $n$ ): a finite window of white noise; $\mathrm{Var}=(\sum c_i^2)\lambda^2$ , and $\gamma(\tau)$ cuts off past lag $n$ . Monic ( $c_0=1$ ) removes the scale redundancy; always stationary.
AR( $n$ ): a regression on the past; Yule–Walker links parameters and covariances both ways. For AR(1), $\gamma(0)=\lambda^2/(1-a^2)$ and $\gamma(\tau)=a^{\tau}\gamma(0)$ — a geometric tail.
ARMA = poles + zeros; covariance tails off (vanishing-covariance property). The three fingerprints — MA cuts off, AR/ARMA tail off — name a process from its covariance.
ARMAX adds an exogenous input via $B(z)u(t-k)$ ; the two-path (input + noise) picture underlies all of identification.

Exam radar. Be able to (i) write Yule–Walker for an AR(1)/AR(2) and solve either direction, (ii) compute an MA variance and covariance from the coefficients, and (iii) name a class from a covariance table. These feed straight into the process-analysis and identification problems.