Linear Regression · Machine Learning

01 · Motivation

Why does this matter?

Pick a number you wish you could predict. House price next month. Tomorrow’s temperature. A patient’s blood-pressure response to a new dose. These are all regression problems: the target is a real number on a continuous scale, and you want to estimate it from a handful of measured inputs.

Out of the whole zoo of regression models, the linear one — a weighted sum of the inputs plus an offset — is the first thing a working data scientist tries and the model every later technique is benchmarked against. Three reasons earn it that status:

It's solvable on paper

Linear regression has a one-formula, closed-form answer — no iterative optimisation, no random seeds, just $\hat{\mathbf{w}} = (\boldsymbol{\Phi}^\top\boldsymbol{\Phi})^{-1}\boldsymbol{\Phi}^\top\mathbf{t}$ . That clarity makes it the perfect lab specimen.

It teaches the whole toolbox

Loss functions, gradient descent, the bias–variance trade-off, regularisation, MAP and MLE, the geometry of projection — every one of these shows up here in its cleanest form. Master it and you’ve touched 70% of supervised learning.

It's a real workhorse

Many physical, biological and economic relationships are locally linear. And when they aren’t, one small twist — basis functions — extends the model to arbitrary curves while keeping solvability.

why

The reason linear models keep winning

Across decades of empirical studies, linear models with thoughtful features beat fancier alternatives roughly half the time. They train in milliseconds, extrapolate predictably, every coefficient has a meaning, and they fail loudly when their assumptions break. Reach for the simple thing first; reach for the deep network only when you can show the simple thing is not enough.

This chapter is the gateway to every supervised model that follows. Logistic regression (Ch. 3) is linear regression with a sigmoid. SVMs (Ch. 7) swap the loss. Kernel methods (Ch. 6) do it in an infinite feature space. Neural networks stack linear-plus-non-linearity. The line you fit here is the first vertebra of a long spine.

02 · Intuition

The idea in plain language

Imagine a cloud of points on a 2-D plot. The horizontal axis is something you can measure cheaply — say, the size of a flat in square metres. The vertical axis is something you’d like to predict — its sale price. You squint, notice the cloud slopes up to the right, and reach for a ruler. Linear regression is the maths of choosing where to put the ruler.

Two knobs control any straight line:

The intercept $w_0$ — where the line crosses the vertical axis. Lift it, the whole line shifts up.
The slope $w_1$ — how steeply it climbs as you move right. Steepen it, the line rotates.

Two unknowns, infinitely many candidate lines. The job is to pick the one line that “agrees with the cloud” the most — so we must be precise about what disagreement means.

Why we measure error vertically

For each training point, look straight up or down to the line. The length of that vertical segment is the residual $\varepsilon_n = t_n - \hat h(x_n)$ — what the model got wrong, in the units of the output. We square each residual before summing, for three reasons that compound:

Sign cancellation

A point 5 above the line shouldn’t cancel one 5 below. Squaring forces every miss to count as positive.

Big mistakes matter more

Squaring hurts four times as much for a residual of 2 as for one of 1 — the model is forced to pay attention to large errors.

Differentiability

Squared error is smooth everywhere. Set the gradient to zero, solve a linear system, done. (Absolute error has a kink at zero — Ch. 7.)

tip

A useful image: the shortest vertical ropes

Imagine an elastic rope from every training point straight down (or up) to your candidate line. The shorter the ropes on average, the better the fit. Least squares says: pick the line so the sum of squared rope lengths is as small as possible. That phrase is the whole pedagogy of this chapter.

”Linear” — but linear in what?

Here is the subtlety that catches every student once. A model can be linear in the inputs, in the parameters, or both. In machine learning, “linear regression” means linear in the parameters. Linear in the inputs is a bonus, not a requirement.

Linear in w → counts as linear regression

The output is a weighted sum of known functions of $x$ ; the weights enter linearly and we solve with a single matrix formula.

$y = w_0 + w_1 x + w_2 x^2$
$y = w_0 + w_1 \sin x + w_2 e^{-x^2}$
$y = w_0 + w_1\,\phi_1(\mathbf{x}) + w_2\,\phi_2(\mathbf{x})$

Non-linear in w → NOT linear regression

Weights appear inside non-linear functions or multiplied together. No closed form; you must optimise iteratively (and may find only local minima).

$y = w_0 + e^{w_1 x}$
$y = w_0 \sin(w_1 x)$
$y = \sigma(w_0 + w_1 x)$ (this is Ch. 3)

Once you accept this distinction, the world of “linear” regression becomes surprisingly large. Polynomials, splines, radial-basis networks, Fourier expansions — all of them are linear regression with cleverer features. We formalise this with basis functions next.

03 · Formalism

Definitions and equations

Now we anchor the intuition in notation. Every symbol below re-appears in chapters 3, 6, and 7 — invest the minute to make them stick.

x: an input, a column vector in $\mathbb{R}^{D}$ . For an intercept we tack a constant 1 on top: the augmented input $\mathbf{x} = (1, x_1, \dots, x_{D-1})^\top$ .
t: the target, a real number. The training set stacks $N$ of them in $\mathbf{t} = (t_1, \dots, t_N)^\top$ .
w: the parameter vector — one weight per feature. $w_0$ is the offset, the rest are slopes.
y: the prediction $y(\mathbf{x}, \mathbf{w})$ the model emits for input $\mathbf{x}$ .
ε: the residual on example $n$ : $\varepsilon_n = t_n - y(\mathbf{x}_n, \mathbf{w})$ .
L: the loss — one number scoring how bad $\mathbf{w}$ is on the training set.

The linear model

Plain linear regression writes the prediction as a weighted sum of the input coordinates plus an offset:

Linear in x

y(\mathbf{x}, \mathbf{w}) \;=\; w_0 + \sum_{j=1}^{D-1} w_j\, x_j \;=\; \mathbf{w}^\top \mathbf{x}

With the augmented input $\mathbf{x} = (1, x_1, \dots, x_{D-1})^\top$ the offset folds into the dot product — neat for the maths, easy to forget when you write code.

Buying non-linearity with basis functions

Replace the raw coordinates $x_j$ with hand-chosen transformations $\phi_j(\mathbf{x})$ — features that may be wildly non-linear in $\mathbf{x}$ but are still fed into a linear combination:

Linear in w

y(\mathbf{x}, \mathbf{w}) \;=\; w_0 + \sum_{j=1}^{M-1} w_j\,\phi_j(\mathbf{x}) \;=\; \mathbf{w}^\top \boldsymbol{\phi}(\mathbf{x})

Convention: $\phi_0(\mathbf{x}) \equiv 1$ so $w_0$ is still the offset. $M$ is the number of basis functions (including the constant); it controls the size of the hypothesis space.

Three families show up over and over again:

Family	$\phi_j(x)$	Character
Polynomial	$x^{j}$	Each basis adds a degree. Cheap and powerful but global — a wiggle near $x=0$ changes the fit near $x=10$ . Numerically fragile at high degree.
Gaussian	$\exp\!\bigl(-\tfrac{(x-\mu_j)^2}{2s^2}\bigr)$	Bumps centred at $\mu_j$ . Local — each weight affects only its neighbourhood. Width $s$ sets smoothness; needs feature scaling.
Sigmoidal	$\sigma\!\bigl(\tfrac{x-\mu_j}{s}\bigr)$	Smooth steps. Useful for threshold/saturation behaviour; the building block of logistic regression and neural nets.

key

The basis-function move, in one sentence

Choose features wisely and a linear model can fit almost anything. The heavy lifting moves from “what kind of curve?” to “what kind of $\boldsymbol{\phi}$ ?”. The closed-form solution below treats every basis identically — once $\boldsymbol{\Phi}$ is built, the maths is the same.

The design matrix $\boldsymbol{\Phi}$

Stack one row per training example, one column per basis function. This is the design matrix, the object you’ll see in every derivation from now on:

Design matrix

\boldsymbol{\Phi} \;=\; \begin{pmatrix} \phi_0(\mathbf{x}_1) & \cdots & \phi_{M-1}(\mathbf{x}_1) \\ \vdots & & \vdots \\ \phi_0(\mathbf{x}_N) & \cdots & \phi_{M-1}(\mathbf{x}_N) \end{pmatrix} \in \mathbb{R}^{N \times M}

All $N$ predictions in one shot: $\hat{\mathbf{t}} = \boldsymbol{\Phi}\mathbf{w}$ . The first column is all ones (because $\phi_0 \equiv 1$ ), encoding the offset.

The squared-error loss

Pack the residuals into a vector $\boldsymbol{\varepsilon} = \mathbf{t} - \boldsymbol{\Phi}\mathbf{w}$ ; the loss is the squared $\ell_2$ -norm of that vector:

RSS

L(\mathbf{w}) \;=\; \tfrac{1}{2}\sum_{n=1}^{N}\bigl(t_n - \mathbf{w}^\top \boldsymbol{\phi}(\mathbf{x}_n)\bigr)^2 \;=\; \tfrac{1}{2}\,\|\mathbf{t} - \boldsymbol{\Phi}\mathbf{w}\|_2^{\,2}

The $\tfrac{1}{2}$ is cosmetic — it cancels when you differentiate. $L$ is also the residual sum of squares (RSS); dividing by $N$ gives the mean squared error (MSE).

Setting the gradient $\nabla_{\mathbf{w}} L = \boldsymbol{\Phi}^\top(\boldsymbol{\Phi}\mathbf{w} - \mathbf{t})$ to zero gives the normal equations and the closed-form OLS estimator:

OLS

\boldsymbol{\Phi}^\top \boldsymbol{\Phi}\,\hat{\mathbf{w}} = \boldsymbol{\Phi}^\top \mathbf{t} \quad\Longrightarrow\quad \hat{\mathbf{w}} \;=\; (\boldsymbol{\Phi}^\top \boldsymbol{\Phi})^{-1}\boldsymbol{\Phi}^\top \mathbf{t}

The Hessian $\boldsymbol{\Phi}^\top\boldsymbol{\Phi}$ is positive semi-definite, so this critical point is a global minimum — unique when $\boldsymbol{\Phi}$ has full column rank.

Why squared error? The probabilistic answer

Beyond the three intuitive reasons in §2, the squared loss has a deep justification. Suppose the target is a deterministic function plus Gaussian noise of fixed variance:

t \;=\; \mathbf{w}^\top \boldsymbol{\phi}(\mathbf{x}) + \varepsilon, \qquad \varepsilon \sim \mathcal{N}(0, \sigma^2).

Then $t \mid \mathbf{x}, \mathbf{w}$ is itself Gaussian, and with $N$ i.i.d. examples the log-likelihood is

\ln p(\mathbf{t}\mid\mathbf{X},\mathbf{w},\sigma^2) \;=\; -\tfrac{N}{2}\ln(2\pi\sigma^2) \;-\; \tfrac{1}{2\sigma^2}\underbrace{\sum_{n=1}^{N}\bigl(t_n - \mathbf{w}^\top\boldsymbol{\phi}(\mathbf{x}_n)\bigr)^2}_{\text{RSS}(\mathbf{w})}.

=

MLE = least squares, exactly

The first term has no $\mathbf{w}$ in it; the second is $-\tfrac{1}{\sigma^2}$ times the RSS. So maximising the log-likelihood over $\mathbf{w}$ is identical to minimising the RSS. Squared error is the negative log-likelihood of a Gaussian noise model, up to constants. When you fit a line by minimising vertical squared distances, you are assuming the noise is symmetric, Gaussian, and the same size everywhere.

Optional depth The unbiased noise-variance estimate, and the Minkowski loss family

Differentiating the log-likelihood with respect to $\sigma^2$ gives the MLE of the noise variance as the mean squared residual. The bias-corrected version divides by $N - M$ instead of $N$ , because $M$ degrees of freedom were “used up” fitting the weights:

\hat\sigma^2 \;=\; \frac{1}{N - M}\sum_{n=1}^{N}\bigl(t_n - \hat{\mathbf{w}}^\top\boldsymbol{\phi}(\mathbf{x}_n)\bigr)^2.

If $N = M$ the model fits perfectly and $\hat\sigma^2$ is undefined — a structural warning, not a bug.

Squared error is one member of a family $L(t,y) = |t-y|^q$ . Different $q$ encode different noise models:

$q$	Loss	Optimal predictor at $\mathbf{x}$	Implicit noise
2	squared error	conditional mean $\mathbb{E}[t\mid\mathbf{x}]$	Gaussian
1	absolute error	conditional median	Laplace
$\to 0$	0/1 hit	conditional mode	—

Use $q=1$ when outliers are real and you don’t want a few extreme points dragging the fit. The rest of this chapter sticks with $q=2$ .

When the closed form is too expensive: gradient descent

The OLS formula inverts an $M \times M$ matrix at cost $O(NM^2 + M^3)$ . For thousands of features, or streaming data that never fits in memory, the fallback is iterative gradient descent:

Gradient descent

\mathbf{w}^{(k+1)} \;=\; \mathbf{w}^{(k)} \;-\; \alpha^{(k)}\,\nabla_{\mathbf{w}} L\bigl(\mathbf{w}^{(k)}\bigr), \qquad \nabla_{\mathbf{w}} L = \boldsymbol{\Phi}^\top(\boldsymbol{\Phi}\mathbf{w} - \mathbf{t}).

$\alpha^{(k)}$ is the learning rate. Too small and convergence crawls; too large and the iterates overshoot and diverge. Hands-on lab #2 lets you feel both failure modes.

Batch gradient descent uses the full sum over $N$ samples each step ( $O(NM)$ ). Stochastic GD (SGD) uses one sample (or a mini-batch) per step ( $O(M)$ , works online) at the price of a jittery path. SGD converges if the learning-rate schedule satisfies the Robbins–Monro conditions $\sum_k \alpha^{(k)} = \infty$ and $\sum_k (\alpha^{(k)})^2 < \infty$ — large enough to travel anywhere, shrinking fast enough that the noise settles. For the convex quadratic loss of linear regression, gradient descent reaches the same $\hat{\mathbf{w}}_\text{OLS}$ the closed form gives: a different route to the same answer, and the route every later model is forced to take.

04 · Worked example

Closed-form OLS on four points, by hand

Time to make every symbol concrete. We fit a straight line to four points with the closed-form formula, doing the matrix arithmetic step-by-step. This is the derivation the exam asks you to reproduce.

Worked example OLS by hand on four points

1 · The dataset

Four flat sizes (tens of m²) and their measured rents (hundreds of €):

$n$	$x_n$	$t_n$
1	1	2
2	2	3
3	3	5
4	4	6

We model rent as a line $y(x,\mathbf{w}) = w_0 + w_1 x$ , with the simplest basis $\phi_0(x)=1,\;\phi_1(x)=x$ .

2 · Build the design matrix and target vector

\boldsymbol{\Phi} \;=\; \begin{pmatrix} 1 & 1 \\ 1 & 2 \\ 1 & 3 \\ 1 & 4 \end{pmatrix}, \qquad \mathbf{t} \;=\; \begin{pmatrix} 2 \\ 3 \\ 5 \\ 6 \end{pmatrix}

First column is the ones for the offset; second column is the $x$ values themselves.

3 · Run the normal-equation pipeline

Compute $\boldsymbol{\Phi}^\top\boldsymbol{\Phi}$ — a $2\times2$ of column dot-products — and $\boldsymbol{\Phi}^\top\mathbf{t}$ :

\boldsymbol{\Phi}^\top\boldsymbol{\Phi} = \begin{pmatrix} N & \sum x_n \\ \sum x_n & \sum x_n^2 \end{pmatrix} = \begin{pmatrix} 4 & 10 \\ 10 & 30 \end{pmatrix}, \qquad \boldsymbol{\Phi}^\top\mathbf{t} = \begin{pmatrix} \sum t_n \\ \sum x_n t_n \end{pmatrix} = \begin{pmatrix} 16 \\ 47 \end{pmatrix}

Invert the $2\times2$ — determinant $\det = (4)(30) - (10)(10) = 20$ :

(\boldsymbol{\Phi}^\top\boldsymbol{\Phi})^{-1} = \tfrac{1}{20}\begin{pmatrix} 30 & -10 \\ -10 & 4 \end{pmatrix} = \begin{pmatrix} 1.5 & -0.5 \\ -0.5 & 0.2 \end{pmatrix}

Multiply through:

\hat{\mathbf{w}} = \begin{pmatrix} 1.5 & -0.5 \\ -0.5 & 0.2 \end{pmatrix}\begin{pmatrix} 16 \\ 47 \end{pmatrix} = \begin{pmatrix} 0.5 \\ 1.4 \end{pmatrix}

So $\hat w_0 = 0.5$ , $\hat w_1 = 1.4$ , and the best-fit line is $\hat y(x) = 0.5 + 1.4\,x$ .

4 · Sanity-check the residuals

Plug the four inputs back in and read the residuals:

$x_n$	$t_n$	$\hat y_n$	$\varepsilon_n = t_n - \hat y_n$
1	2	1.9	$+0.1$
2	3	3.3	$-0.3$
3	5	4.7	$+0.3$
4	6	6.1	$-0.1$

Two checks pop out for free, both worth memorising:

The residuals sum to zero: $0.1 - 0.3 + 0.3 - 0.1 = 0$ . Whenever the model has an intercept, this is automatic.
The residuals are uncorrelated with each feature: $1(0.1) + 2(-0.3) + 3(0.3) + 4(-0.1) = 0$ . This is exactly the optimality condition $\boldsymbol{\Phi}^\top(\mathbf{t} - \boldsymbol{\Phi}\hat{\mathbf{w}}) = \mathbf{0}$ .

The total loss is $L = \tfrac{1}{2}(0.1^2 + 0.3^2 + 0.3^2 + 0.1^2) = 0.10$ .

map

Map the example back onto the formalism

$N = 4$ training samples; $M = 2$ basis functions ( $1$ and $x$ ).
$\boldsymbol{\Phi} \in \mathbb{R}^{4\times 2}$ , $\mathbf{t} \in \mathbb{R}^{4}$ , $\mathbf{w} \in \mathbb{R}^{2}$ .
Closed form: $\hat{\mathbf{w}} = (0.5,\,1.4)^\top$ .
Residual identities $\sum \varepsilon_n = 0$ and $\sum x_n \varepsilon_n = 0$ are the optimality conditions, written out.

05 · Generalization

Underfitting, overfitting, and why training fit isn’t enough

We have a closed form that minimises RSS on the training data. The temptation is to declare victory and pick whatever hits the training points hardest. That temptation is the single biggest trap in supervised learning, and it shows up first and clearest in linear regression.

Fit polynomials of growing degree $M$ to data from a smooth curve plus noise:

$M = 1$ (a line). Too rigid to capture the curve — both training and unseen error are large. Underfitting: high bias.
$M = 3$ (a cubic). Flexible enough to bend, simple enough not to chase noise. Both errors drop.
$M = 9$ on 10 points. Passes through every training point, training RSS $\approx 0$ — but it wiggles wildly between points and predicts garbage off-training. Overfitting: it memorised the noise.

Training error tells you only how well the model learned what it has already seen. To estimate performance on the next input, measure error on a held-out test set. Test error follows a characteristic U-shape: high bias on the left, high variance on the right, the sweet spot in the middle.

!

A tell-tale fingerprint of overfitting

When linear regression with too many basis functions overfits, the fitted weights become enormous and opposite-signed, so big positive and negative contributions cancel at the training points and explode between them. Bishop’s classic degree-9 fit to 10 sinusoidal points produces coefficients of order $10^5$ . If you ever see weights orders of magnitude larger than your data, you have an overfitting problem — and ridge regression is the first thing to try.

The natural response — penalise large weights so the optimiser prefers smoother solutions — is exactly regularisation, and it produces the next two estimators we study (ridge and lasso). You can watch all of this happen in the hands-on labs below.

06 · Visual explanation

The geometry of least squares

The formulas hide a story that becomes obvious once you draw it.

The prediction as a projection

Lift your eyes from the scatter plot. The $N$ target values stack into a single vector $\mathbf{t}$ in $\mathbb{R}^N$ — one dimension per data point. Each column of $\boldsymbol{\Phi}$ is also an $N$ -dimensional vector; together the columns span an $M$ -dimensional subspace $S$ . Every prediction $\boldsymbol{\Phi}\mathbf{w}$ the model can produce lives in $S$ , no matter how you choose $\mathbf{w}$ .

With $\mathbf{t}$ hovering off the plane, the closest reachable point is the orthogonal projection $\hat{\mathbf{t}} = \boldsymbol{\Phi}\hat{\mathbf{w}}$ . The residual $\mathbf{t} - \hat{\mathbf{t}}$ sticks out perpendicular to $S$ — exactly the condition $\boldsymbol{\Phi}^\top(\mathbf{t} - \boldsymbol{\Phi}\hat{\mathbf{w}}) = \mathbf{0}$ we got by setting the gradient to zero. The matrix that performs the projection is the hat matrix $\mathbf{H} = \boldsymbol{\Phi}(\boldsymbol{\Phi}^\top\boldsymbol{\Phi})^{-1}\boldsymbol{\Phi}^\top$ , with $\hat{\mathbf{t}} = \mathbf{H}\mathbf{t}$ — symmetric, idempotent, and its trace equals the effective number of fitted parameters.

view

Two pictures, same maths

The scatter-plot view (“vertical residuals, sum of squares”) and the subspace view (“orthogonal projection in $\mathbb{R}^N$ ”) are literally the same calculation. The first lives in input space and is friendly; the second lives in sample space and is profound — it generalises immediately to ridge, kernel methods, and the Gauss–Markov theorem.

Ridge vs lasso — the constraint geometry

Regularised least squares is “minimise $L$ while the weight vector stays inside a small region of parameter space.” For ridge that region is a ball; for lasso, a diamond. The shape decides the answer.

The elliptical contours are the loss — each ellipse is “all $\mathbf{w}$ with the same training error”. The regularised optimum is where the smallest ellipse first touches the constraint region. Ridge’s smooth ball is usually touched at a point with all components non-zero, simply shrunk. The lasso diamond has corners; the ellipse usually touches at a corner, sending some components to exactly zero — which is why lasso performs variable selection and ridge does not. In formulas the two regularisers differ by one exponent:

L_{\text{ridge}}(\mathbf{w}) = \tfrac{1}{2}\|\mathbf{t} - \boldsymbol{\Phi}\mathbf{w}\|_2^2 + \tfrac{\lambda}{2}\|\mathbf{w}\|_2^2, \qquad L_{\text{lasso}}(\mathbf{w}) = \tfrac{1}{2}\|\mathbf{t} - \boldsymbol{\Phi}\mathbf{w}\|_2^2 + \tfrac{\lambda}{2}\|\mathbf{w}\|_1.

As the constraint shrinks (higher $\lambda$ ), predictions become more biased but less sensitive to the particular training sample (lower variance) — the bias–variance trade-off, seen here in its cleanest setting and the central story of Ch. 4.

07 · The Bayesian view

Treating the weights as a probability distribution

Everything so far produced a single best vector $\hat{\mathbf{w}}$ . The Bayesian view refuses to commit to one answer: it carries an entire distribution over the weights — narrow when the data are informative, wide when they are not — and uses it for both prediction and honest uncertainty.

Before seeing data we encode beliefs about plausible weights as a prior $p(\mathbf{w})$ . After observing $\mathcal{D}$ , Bayes’ rule gives the posterior:

\underbrace{p(\mathbf{w}\mid\mathcal{D})}_{\text{posterior}} \;\propto\; \underbrace{p(\mathcal{D}\mid\mathbf{w})}_{\text{likelihood}}\;\underbrace{p(\mathbf{w})}_{\text{prior}}.

For Gaussian noise the likelihood is Gaussian in $\mathbf{w}$ ; choose a Gaussian prior $\mathcal{N}(\mathbf{w}_0, \mathbf{S}_0)$ and the posterior is again Gaussian (the prior is conjugate):

Posterior

\mathbf{S}_N^{-1} = \mathbf{S}_0^{-1} + \tfrac{1}{\sigma^2}\boldsymbol{\Phi}^\top\boldsymbol{\Phi}, \qquad \mathbf{w}_N = \mathbf{S}_N\!\left(\mathbf{S}_0^{-1}\mathbf{w}_0 + \tfrac{1}{\sigma^2}\boldsymbol{\Phi}^\top\mathbf{t}\right).

Read $\mathbf{S}_N^{-1}$ as “total information = prior information + data information”. The posterior mean is a precision-weighted blend of prior mean and data estimate — and the posterior can serve as the prior for the next batch, so Bayesian updating is naturally online.

Connection to ridge regression

The mode of a Gaussian is its mean, so the MAP estimate is $\mathbf{w}_N$ . Set a zero-mean isotropic prior $\mathbf{w}_0 = \mathbf{0}$ , $\mathbf{S}_0 = \tau^2\mathbf{I}$ and it simplifies to

\mathbf{w}_N = \Bigl(\tfrac{\sigma^2}{\tau^2}\mathbf{I} + \boldsymbol{\Phi}^\top\boldsymbol{\Phi}\Bigr)^{-1}\boldsymbol{\Phi}^\top\mathbf{t}.

Compare with ridge: identical, with $\lambda = \sigma^2/\tau^2$ . Ridge regression is MAP estimation under a Gaussian prior on the weights. A tight prior means strong regularisation; a vague prior ( $\tau^2 \to \infty$ ) recovers OLS.

Predictive distribution: prediction with error bars

For a new input $\mathbf{x}^\star$ , instead of plugging in one $\hat{\mathbf{w}}$ we integrate over the whole posterior. The result is again Gaussian:

Posterior predictive

\sigma_N^2(\mathbf{x}^\star) \;=\; \underbrace{\sigma^2}_{\text{irreducible noise}} \;+\; \underbrace{\boldsymbol{\phi}(\mathbf{x}^\star)^\top \mathbf{S}_N\, \boldsymbol{\phi}(\mathbf{x}^\star)}_{\text{uncertainty about the weights}}.

Two sources of uncertainty, cleanly separated. The first never disappears — even a perfect model cannot predict pure noise. The second shrinks toward zero as data accumulate. Predictions far from the training inputs get wider error bars, exactly the behaviour you want.

why

Why care about the Bayesian view?

Three reasons. It gives a principled, automatic way to regularise (the prior is the regulariser); it returns honest uncertainty alongside every prediction; and it unifies the chapter — OLS is MLE, ridge is MAP, and full Bayesian prediction generalises both. The price is more integration — but for linear-Gaussian models, every integral is closed form.

08 · Hands-on

Try it yourself

Four labs, each tuned to drill in one concept that students usually need to see move before it sticks. Push the controls, watch the numbers, then read the takeaway.

Hands-on 1

Watch the line settle on noisy data

The true relationship is t = −0.3 + 0.9 x. Each Resampledraws fresh training points (orange) from that line plus Gaussian noise, and the least-squares line (gold) is fit on the spot. Raise the noise to see the fit wobble; raise the sample size to see it lock back in.

noise σ0.25

samples N15

ŵ₀ (intercept)

-0.377

ŵ₁ (slope)

1.060

Train MSE

0.072

Σ residual

-0.00

Try thisSet σ = 0.10, N = 10 and resample a few times — the slope jitters around 0.9 by a few hundredths. Crank σ to 0.60 and it might land at 0.6 or 1.2 on different draws. Now push N to 60: the slope locks back near 0.9 even with high noise.

TakeawayThe fit is a random object — every resample gives a slightly different ŵ. Its variability shrinks with more data and grows with noise, exactly as Var(ŵ) = (ΦᵀΦ)⁻¹σ² predicts. The intercept also forces Σ residual ≈ 0 on every draw.

Hands-on 2

Step gradient descent across the loss surface

The closed form is instant, but real problems descend iteratively: w⁽ᵏ⁺¹⁾ = w⁽ᵏ⁾ − α ∇L(w⁽ᵏ⁾). The left panel is the loss L(w₀,w₁) (bright = low); the right shows the current line on the data. Tune α to feel under-shoot, the sweet spot, and over-shoot.

learning rate α0.080

iter 0

Loss contours · trajectory

Current line on the data

w₀

1.500

w₁

-0.400

L(w)

0.177

‖∇L‖

0.472

Try thisStep with α = 0.08 — the path spirals neatly into the green optimum. Reset, set α = 0.40 and step: the iterates jump across the valley and the loss grows. Drop α to 0.005 and it crawls, dozens of steps to reach the bottom.

TakeawayDescent on a quadratic loss is a tug between aggression (large α reaches the bottom fast) and stability (too large and it diverges). The closed form short-circuits all of this — but most real losses are not quadratic, so the iterative route is the one every later model takes.

Hands-on 3

Linear → polynomial → Gaussian basis

The data come from a smooth non-linear curve (blue). Switch the basis family and the number of basis functions M. The fit stays linear in w— the closed-form formula never changes — but the shapes it can take do.

M (basis count)5

Train MSE

0.084

Test MSE

0.086

Parameters M

5

Try thisWith Linear, M is locked at 2 and the fit cannot bend. Switch to Polynomial and push M to 12 — train MSE drops but the curve wiggles at the edges and test MSE jumps. Now try Gaussian at M = 8: the curve tracks the data smoothly because each bump is local.

TakeawayThe basis is a prior over plausible curves. Polynomials assume one global formula; Gaussians assume a sum of local bumps. "Linear regression" with a clever basis covers a huge zoo of shapes — without ever leaving the closed-form world.

Hands-on 4

Ridge regression — tame an overfit polynomial

We fit a degree-15 polynomial to 12 noisy points — far more parameters than data. Slide log₁₀ λ and watch ŵ = (λI + ΦᵀΦ)⁻¹Φᵀt reshape the fit, collapse the weight norm ‖w‖², and trade train error for test error.

log₁₀ λ10^-12

overfitting · weights explode

Train MSE

0.035

Test MSE

0.099

‖w‖²

6.4e+6

eff. d.o.f.

9.8

Try thisStart at log₁₀ λ = −12: the curve hits every point but explodes between them and ‖w‖² is enormous. Drag rightward — the curve smooths, the weight norm collapses, and test MSE first falls(you bought generalisation) then rises (you over-penalised). The minimum test MSE is the sweet spot cross-validation finds in Ch. 4.

TakeawayRidge does not delete parameters; it shrinks them proportionally. The model is still degree-15 but its effective degrees of freedom are small — a tunable λ knob riding the bias–variance dial.

09 · Exam intel

What the exam actually tests

Linear regression carries a heavy exam load — derivations are short, definite, and easy to grade. Five question shapes appear essentially every year.

Q1

Derive the closed-form OLS estimator

Start from $L(\mathbf{w}) = \tfrac{1}{2}(\mathbf{t} - \boldsymbol{\Phi}\mathbf{w})^\top(\mathbf{t} - \boldsymbol{\Phi}\mathbf{w})$ . Take the gradient $\nabla_{\mathbf{w}} L = -\boldsymbol{\Phi}^\top\mathbf{t} + \boldsymbol{\Phi}^\top\boldsymbol{\Phi}\mathbf{w}$ , set it to zero for the normal equations $\boldsymbol{\Phi}^\top\boldsymbol{\Phi}\mathbf{w} = \boldsymbol{\Phi}^\top\mathbf{t}$ , and conclude $\hat{\mathbf{w}} = (\boldsymbol{\Phi}^\top\boldsymbol{\Phi})^{-1}\boldsymbol{\Phi}^\top\mathbf{t}$ . For full marks, note the Hessian is positive semi-definite, so the critical point is a global minimum.

Q2

Show that MLE = LS under Gaussian noise

Assume $t_n = \mathbf{w}^\top\boldsymbol{\phi}(\mathbf{x}_n) + \varepsilon_n$ with $\varepsilon_n \sim \mathcal{N}(0,\sigma^2)$ i.i.d. Write the log-likelihood; the term in $\mathbf{w}$ is $-\tfrac{1}{2\sigma^2}\,\text{RSS}(\mathbf{w})$ , so maximising it over $\mathbf{w}$ minimises the RSS. One-sentence summary: “squared error is the negative log-likelihood of a Gaussian noise model, up to constants.”

Q3

Derive the ridge estimator

The penalised loss $\tfrac{1}{2}\|\mathbf{t} - \boldsymbol{\Phi}\mathbf{w}\|^2 + \tfrac{\lambda}{2}\|\mathbf{w}\|^2$ has gradient $-\boldsymbol{\Phi}^\top\mathbf{t} + \boldsymbol{\Phi}^\top\boldsymbol{\Phi}\mathbf{w} + \lambda\mathbf{w}$ ; setting it to zero gives $\hat{\mathbf{w}}_\text{ridge} = (\lambda\mathbf{I} + \boldsymbol{\Phi}^\top\boldsymbol{\Phi})^{-1}\boldsymbol{\Phi}^\top\mathbf{t}$ . Mention: (i) $\lambda\mathbf{I} + \boldsymbol{\Phi}^\top\boldsymbol{\Phi}$ is positive definite for any $\lambda > 0$ , so it’s always invertible — even with collinear features; (ii) ridge is MAP under a Gaussian prior with $\lambda = \sigma^2/\tau^2$ ; (iii) lasso uses $\|\mathbf{w}\|_1$ , has no closed form, and yields sparse solutions.

Q4

State and use the Gauss–Markov theorem

Among linear unbiased estimators of $\mathbf{w}$ , ordinary least squares has the smallest variance, component by component: $\mathrm{Var}(\hat{\mathbf{w}}_\text{OLS}) = (\boldsymbol{\Phi}^\top\boldsymbol{\Phi})^{-1}\sigma^2$ . The catch: “unbiased” is doing real work — a biased estimator like ridge can have smaller mean-squared error. Gauss–Markov is what motivates regularisation as a deliberate bias-for-variance trade.

Q5

Place linear regression on the four ML dichotomies

Dichotomy	Where linear regression sits
Parametric vs Nonparametric	Parametric — fixed $M$ parameters
Frequentist vs Bayesian	Frequentist by default; Bayesian under a prior $p(\mathbf{w})$
Generative vs Discriminative	Discriminative (direct $p(t\mid\mathbf{x})$ or just $y(\mathbf{x},\mathbf{w})$ )
ERM vs SRM	OLS is empirical-risk minimisation; ridge/lasso are structural-risk minimisation

tip

Memorise four formulas and you have the chapter

Linear model: $y(\mathbf{x},\mathbf{w}) = \mathbf{w}^\top\boldsymbol{\phi}(\mathbf{x})$ .
OLS: $\hat{\mathbf{w}} = (\boldsymbol{\Phi}^\top\boldsymbol{\Phi})^{-1}\boldsymbol{\Phi}^\top\mathbf{t}$ .
Ridge: $\hat{\mathbf{w}}_\text{ridge} = (\lambda\mathbf{I} + \boldsymbol{\Phi}^\top\boldsymbol{\Phi})^{-1}\boldsymbol{\Phi}^\top\mathbf{t}$ .
Predictive variance: $\sigma_N^2(\mathbf{x}) = \sigma^2 + \boldsymbol{\phi}(\mathbf{x})^\top\mathbf{S}_N\,\boldsymbol{\phi}(\mathbf{x})$ .

10 · Common mistakes

Where students get this wrong

×

” $y = w_0 + w_1 x^2$ is non-linear regression” It looks bent, but it is linear regression — linear in the parameter vector $(w_0, w_1)$ . The non-linearity lives in the fixed, known basis $\phi_1(x) = x^2$ . The closed-form OLS formula applies untouched. The distinction is “linear in $\mathbf{w}$ ”, not “linear in $x$ ”.

×

Forgetting to scale features before Gaussian or polynomial bases

A Gaussian basis has its bandwidth $s$ baked into the units of $x$ ; a single $s$ cannot fit metres and kilograms at once. Always standardise (zero mean, unit variance) before fitting — same goes for high-degree polynomials, where $x^{10}$ explodes unless $x \in [-1,1]$ .

×

Multicollinearity → 'the OLS formula doesn't work'

When two features are nearly dependent, $\boldsymbol{\Phi}^\top\boldsymbol{\Phi}$ becomes near-singular, and inverting it gives wild, opposite-signed coefficients that cancel on training data but blow up on test inputs. Three honest responses: drop one offender, combine them (PCA), or regularise with ridge — $\lambda\mathbf{I} + \boldsymbol{\Phi}^\top\boldsymbol{\Phi}$ is always invertible for $\lambda > 0$ .

×

Maximising R² as the goal

$R^2 = 1 - \text{RSS}/\text{TSS}$ measures explained training variance and is monotone in the number of features — adding a useless predictor never lowers it. A model with $R^2 = 0.99$ on training and catastrophic test error is overfitting, not winning. Use held-out error (Ch. 4), not $R^2$ .

×

'Ridge eliminates parameters'

It doesn’t. Ridge shrinks coefficients toward zero but rarely makes them exactly zero. For sparse, interpretable models with a built-in feature selector you want lasso ( $\ell_1$ penalty) — the diamond geometry in §6 is the visual reason.

×

'More basis functions always improve the fit'

Training error never rises as $M$ grows; test error follows the U-shape — low $M$ underfits, high $M$ overfits. Bishop’s $M=9$ coefficients reach $\pm 10^5$ and predict garbage off-training: a textbook case of why a bigger $\mathcal{H}$ without regularisation is a trap.

×

Confusing 'noise in t' with 'noise in x'

Ordinary linear regression assumes the inputs $x_n$ are known exactly and the target $t_n$ is the noisy thing — that asymmetry is why we measure error vertically, not perpendicular to the line. If the inputs are themselves noisy, OLS is biased; you want errors-in-variables or total least squares (out of scope).

11 · Self-check

Can you answer these?

Six short questions that mirror how the chapter gets tested. Click an option for instant feedback.

Which of the following models is linear regression?

You have N = 200 samples and M = 5 basis functions. What shape is the matrix ΦᵀΦ?

Under what assumption is the least-squares solution identical to the maximum-likelihood estimator?

You fit a degree-15 polynomial: training MSE is near zero but test MSE is much higher. Which intervention is most likely to help?

You place a zero-mean isotropic Gaussian prior N(0, τ²I) on the weights and compute the MAP estimate under Gaussian noise with variance σ². What do you get?

Training MSE is 0.02 and test MSE is 0.18 on a model with M = 12 basis functions and N = 15 samples. What is most likely happening?

12 · Recap

One-screen summary

Chapter 02 — load-bearing ideas

Least squares fits the line that minimises vertical squared residuals. Sign-cancellation, outlier sensitivity, and differentiability are the three reasons we square.
“Linear” means linear in $\mathbf{w}$ , not in $\mathbf{x}$ . Basis functions $\boldsymbol{\phi}(\mathbf{x})$ let the same algorithm fit curves: polynomial (global), Gaussian (local bumps), sigmoidal (smooth steps).
The OLS closed form is $\hat{\mathbf{w}} = (\boldsymbol{\Phi}^\top\boldsymbol{\Phi})^{-1}\boldsymbol{\Phi}^\top\mathbf{t}$ , derived in three lines from $\nabla L = 0$ , at cost $O(NM^2 + M^3)$ .
Geometric story. $\hat{\mathbf{t}}$ is the orthogonal projection of $\mathbf{t}$ onto the column space of $\boldsymbol{\Phi}$ ; the hat matrix $\mathbf{H}$ does the projecting.
Probabilistic story. Under i.i.d. Gaussian noise, MLE coincides exactly with OLS; squared error is the negative log-likelihood of a Gaussian. Unbiased noise estimate $\hat\sigma^2 = \text{RSS}/(N-M)$ .
Generalisation. Training error always falls with complexity $M$ ; test error is U-shaped. Low $M$ underfits (bias), high $M$ overfits (variance); the right $M$ is picked by held-out validation (Ch. 4).
Regularisation. Ridge $= (\lambda\mathbf{I} + \boldsymbol{\Phi}^\top\boldsymbol{\Phi})^{-1}\boldsymbol{\Phi}^\top\mathbf{t}$ shrinks weights and fixes collinearity; lasso ( $\ell_1$ ) gives sparse solutions with no closed form.
Gradient descent for big data. $\mathbf{w}^{(k+1)} = \mathbf{w}^{(k)} - \alpha\,\nabla L$ . Batch is stable but $O(NM)$ ; SGD is $O(M)$ and converges under Robbins–Monro.
Bayesian linear regression. Gaussian prior + Gaussian likelihood ⇒ Gaussian posterior. MAP equals ridge with $\lambda = \sigma^2/\tau^2$ ; predictive variance separates noise from parameter uncertainty.
Four formulas to memorise: the linear model, the OLS solution, the ridge solution, and the predictive variance.

Why does this matter?

It's solvable on paper

It teaches the whole toolbox

It's a real workhorse

The idea in plain language

Why we measure error vertically

Sign cancellation

Big mistakes matter more

Differentiability

”Linear” — but linear in what?

Linear in w → counts as linear regression

Non-linear in w → NOT linear regression

Definitions and equations

The linear model

Buying non-linearity with basis functions

The design matrix Φ\boldsymbol{\Phi}Φ

The squared-error loss

Why squared error? The probabilistic answer

When the closed form is too expensive: gradient descent

Closed-form OLS on four points, by hand

Underfitting, overfitting, and why training fit isn’t enough

The geometry of least squares

The prediction as a projection

Ridge vs lasso — the constraint geometry

Treating the weights as a probability distribution

Connection to ridge regression

Predictive distribution: prediction with error bars

Try it yourself

Watch the line settle on noisy data

Step gradient descent across the loss surface

Linear → polynomial → Gaussian basis

Ridge regression — tame an overfit polynomial

What the exam actually tests

Where students get this wrong

Can you answer these?

One-screen summary

Chapter 02 — load-bearing ideas

The design matrix $\boldsymbol{\Phi}$