Chapter 03

Neural Networks Training & Overfitting

By the end you can diagnose under- vs over-fitting, choose a validation scheme, and sort every training trick into the two buckets the exam keeps asking about: does it improve GENERALIZATION, or training PERFORMANCE?

Reading: ~60 min Interactive: 3 widgets Source: Goodfellow, Bengio & Courville Ch. 7–8 · Bishop Ch. 5

01 · Generalization

The only error that matters is on unseen data

The problem. A universal approximator can fit any training set — including its noise. But training error is a biased, optimistic estimate of future performance: the model was tuned on exactly that data, and you can find patterns even in random noise. What we actually care about is the generalization error on data the model has never seen.

📈 Underfitting (high bias)

Model too simple to capture the pattern. Training and validation error both high. Fix: more capacity / features, train longer.

📉 Overfitting (high variance)

Model memorises training noise. Training error low, validation error high. Fix: regularise, get more data, simplify the model.

why

Revising for an exam

Fitting a model is like revising. Underfitting = you skimmed and learned nothing. Overfitting = you memorised last year’s answer key word-for-word and are lost the moment the questions change. We want understanding that transfers — low error on the new paper.

Conceptually, the expected test error decomposes into three parts — a useful mental model even though we never compute it directly:

Bias–variance (mental model)

\mathbb{E}[(y - \hat{f}(x))^2] = \underbrace{\text{Bias}^2}_{\text{too simple}} + \underbrace{\text{Variance}}_{\text{too sensitive}} + \underbrace{\sigma^2}_{\text{irreducible}}

Capacity trades bias for variance. The irreducible term $\sigma^2$ is the noise floor — no model can beat it.

Hands-on 1

Training vs validation error

Slide the model capacity (depth / width / training time). The training error keeps falling; the validation error falls, then rises. The growing gap is overfitting, and the bottom of the validation curve is where early stopping should halt.

Capacity4.0

Train error

0.253

Val error

0.320

Gap

0.068

Verdict

Good fit

Try thisPush capacity to the far right: training error crawls toward zero while validation error climbs — the model is memorising noise. Slide back to the green "best" line, where the validation error bottoms out.

TakeawayTraining error always falls with capacity, so it can never tell you when to stop. Only the validation curve — and its minimum — reveals the sweet spot between underfitting and overfitting.

02 · Validation & model selection

Spending your data wisely

The vocabulary is exam-relevant — keep the roles straight:

Training set — learn the parameters (weights).
Validation set — model selection: pick hyperparameters (layers, neurons, $\gamma$ ).
Test set — the final, one-shot assessment. Never tuned against.

Two levels of choice happen here: the parameter level (weights, by gradient descent) and the hyperparameter level (architecture / regularisation, by validation error).

✂️ Hold-out

One train/validation split. Cheap, but the estimate is biased by the particular split.

🔁 K-fold CV

Rotate the held-out fold $K$ times, average the errors. The best trade-off; sometimes beats LOOCV.

🎯 LOOCV

$K = N$ (one sample out). Nearly unbiased, but infeasible with lots of data.

K-fold estimate

\hat{E} = \frac{1}{K}\sum_{k=1}^{K}\hat{e}_k, \qquad \hat{e}_k = \frac{1}{\lvert N_k\rvert}\sum_{n\in N_k} E(x_n\mid w)

Early stopping watches the validation error during training and halts when it starts to rise — the moment the network shifts from learning signal to memorising noise. (In classification, keep splits stratified so class proportions are preserved.)

Trap — preprocess per fold

Compute every preprocessing statistic (mean image, normalisation) on the training fold only — never on the full dataset before cross-validation. Using all the data leaks test information into training (2026 Q7).

03 · Regularization

Constraining the model to generalise

Regularisation trades a little training fit for better test performance by constraining the model’s freedom. These are generalization techniques.

L2 weight decay

Add a penalty on the squared weight magnitude:

L2 objective

\arg\min_w \sum_n (t_n - g(x_n\mid w))^2 + \gamma\sum_q w_q^2

Deep dive Why L2 = a Gaussian prior on the weights (MAP)

Maximum likelihood maximises $P(D\mid w)$ . If instead we put a prior $P(w) \sim \mathcal{N}(0, \sigma_w^2)$ on the weights and do maximum a-posteriori estimation:

\hat{w} = \arg\max_w P(w\mid D) = \arg\max_w P(D\mid w)\,P(w)

taking $-\log$ turns the Gaussian likelihood into the usual sum-of-squares and the Gaussian prior into $\gamma\sum_q w_q^2$ . So “small weights generalise better” is literally the statement “I believe, a priori, the weights are near zero”. The strength $\gamma$ is chosen by cross-validation.

Dropout

During training, each hidden unit is zeroed independently with probability $p$ (a Bernoulli mask). This prevents co-adaptation: no unit can rely on a specific partner being present, so each must learn a feature that is useful on its own.

Dropout (training)

h^{(l)} = f\!\left(W^{(l)}\,(h^{(l-1)} \odot m^{(l)})\right), \quad m^{(l)}_j \sim \mathrm{Be}(p)

key

Dropout is an implicit ensemble

Dropout trains an implicit ensemble of sub-networks across mini-batches. At test time the masks are removed and the outputs are averaged (via weight scaling) — dropout is off during inference.

key

Augmentation lives in Chapter 08

Data augmentation and Mixup are also generalization techniques, but they belong to the data-scarcity toolkit — see Chapter 08. Mixup’s target is the same convex combination of the one-hot labels, not the dominant label (2024 Q7).

04 · Batch Normalization & the big sort

Generalization vs performance

The motivation. As a deep net trains, every layer’s weights change, so the distribution of inputs each later layer sees keeps shifting underneath it — the original paper called this internal covariate shift. Layers waste capacity continually re-adapting to a moving target, and gradients become sensitive to scale. Batch Normalization stabilises this by normalising each channel’s pre-activations over the mini-batch, then applying a learnable scale $\gamma$ and shift $\beta$ so the network can undo the normalisation if that helps:

BatchNorm

y_{i,j} = \gamma_j\,\frac{x_{i,j} - \mu_j}{\sqrt{\sigma_j^2 + \epsilon}} + \beta_j

It is placed after a Conv/FC layer and before the non-linearity. During training it uses the batch’s mean/variance; at test time it uses running averages estimated during training. Parameters: $4C$ total per layer — 2C trainable ( $\gamma, \beta$ ) + 2C non-trainable (running $\mu, \sigma^2$ ). That non-trainable half is what makes model.summary() report non-zero “Non-trainable params”.

Exam critical

BatchNorm’s job is faster, more stable convergence (better gradient flow, higher learning rates, less init sensitivity). It is a PERFORMANCE technique with only a mild regularising side-effect — do not file it with Dropout and L2 under “generalization”.

This sort — does a trick improve generalization (which minimum we prefer) or performance (how fast / stably we reach it)? — is asked in every past exam. Drill it:

Hands-on 2

Generalization or performance?

Every training trick has a job. Generalization techniques change which minimum we prefer; performance techniques change how fast and stably we reach it. Sort each one — this exact match question appears on every past exam.

Dropout

Weight decay (L2)

Early stopping

Data augmentation

Batch Normalization

Momentum

Adam

Xavier / He init

ReLU activation

Skip / shortcut connections

Learning-rate schedule

0 / 0

Try thisThe two everyone gets wrong are BatchNorm and skip connections — they feel like regularisers but are primarily performance (optimization) aids. BatchNorm's regularising effect is only a mild side-benefit.

TakeawayAsk one question of any trick: does it change which solution we end up preferring (generalization) or merely how we get there (performance)? That sort is the whole exam topic.

Exam · 2025 Q1 / 2026 Q1 — match each technique

The same match question recurs every year. The canonical key:

Generalization (changes which minimum we prefer): data augmentation, early stopping, weight decay, dropout.
Performance (changes how fast / stably we get there): Batch Normalization, momentum, Xavier/He init, ReLU, skip / shortcut connections, the learning rate.

The two traps are BatchNorm and skip connections — they feel like regularisers but are primarily optimization aids.

05 · Optimization & initialization

Navigating the loss landscape

Activations: ReLU and friends

Sigmoid/tanh saturate: their derivative is $\le 1$ ( $\le 0.25$ for sigmoid) and near-zero in the tails, so multiplied across layers the gradient vanishes. ReLU $= \max(0,x)$ has derivative 0 or 1 — no vanishing, far faster convergence, sparse activations — at the cost of dying neurons (a unit stuck in the negative region outputs 0 forever, since its gradient is 0). Two fixes keep the negative side alive: Leaky ReLU $= \max(\alpha x, x)$ gives a small fixed negative slope $\alpha$ , while ELU $= x$ for $x \ge 0$ and $\alpha(e^x - 1)$ for $x < 0$ saturates smoothly to $-\alpha$ , pushing the mean activation toward zero (which speeds learning) at a little extra compute.

Weight initialization

Zeros → all gradients identical, nothing learns. Too large → exploding gradients. The goal is to preserve activation variance across layers.

📊 Xavier / Glorot

For tanh/sigmoid: $w \sim \mathcal{N}\!\left(0, \tfrac{2}{n_{in}+n_{out}}\right)$ .

📈 He / Kaiming

For ReLU: $w \sim \mathcal{N}\!\left(0, \tfrac{2}{n_{in}}\right)$ .

Deep dive Where the 1/n in Xavier comes from

For a linear neuron $h_j = \sum_i w_{ji}x_i$ with zero-mean i.i.d. inputs and weights, the output variance is

\mathrm{Var}(h_j) = \sum_{i=1}^{I}\mathrm{Var}(w_{ji})\,\mathrm{Var}(x_i) = I\cdot\mathrm{Var}(w)\,\mathrm{Var}(x)

To keep $\mathrm{Var}(h) = \mathrm{Var}(x)$ (variance neither shrinks nor grows layer-to-layer) we need $I\cdot\mathrm{Var}(w) = 1$ , i.e. $\mathrm{Var}(w) = 1/n_{in}$ . Balancing the forward and backward passes gives Glorot’s $2/(n_{in}+n_{out})$ ; He’s $2/n_{in}$ accounts for ReLU zeroing half the activations.

Optimizers & the learning rate

🏃 Momentum

Accumulate a velocity from past gradients; smooths noise, accelerates consistent directions. Nesterov jumps first, then corrects.

📉 RMSprop / AdaGrad

Per-parameter learning rate from a running average of squared gradients.

⚡ Adam

Momentum (1st moment) + RMSprop (2nd moment). The default starting point.

Deep dive The adaptive learning-rate family, in order

Each method fixes the previous one’s weakness — the lineage is the easiest way to remember them:

Rprop — uses only the sign of the gradient, with a per-weight step grown/shrunk by whether the sign stayed the same. Robust full-batch, but breaks with mini-batches (signs too noisy).
AdaGrad — divides the step by $\sqrt{\text{sum of past squared gradients}}$ . Big steps for rare features, but the accumulator only grows, so the effective rate decays to zero and learning stalls.
RMSprop — replaces AdaGrad’s growing sum with an exponential moving average of squared gradients, so the rate no longer dies. (The mini-batch-friendly Rprop.)
AdaDelta — like RMSprop but also removes the need to pick a global rate, using a running average of past updates for the numerator.
Adam — RMSprop’s second moment plus a momentum first moment, both bias-corrected. The robust default.

The learning rate is the single most important hyperparameter. Too high → oscillation/divergence; too low → painfully slow but stable. Standard practice is to decay it high → low; for fine-tuning a pre-trained model use about $1/10$ of the original rate (Chapter 08).

Trap — the learning rate

The learning rate is not required to change monotonically, and a low rate does not harm ReLU networks. You start relatively high and reduce — not the other way round (2026 Q2).

Hands-on 3

SGD vs Momentum vs Adam

Roll an optimizer down a bumpy 1-D loss from the same start. Plain SGD tends to stall in the first dip it meets; Momentum and Adam carry velocity through it toward a deeper minimum. Tune the learning rate and step count.

Learning rate0.30

Steps30

Final x

3.30

Final loss

0.780

Global min?

Stuck

Try thisWith SGD at a small learning rate, watch it settle in the nearest dip. Switch to Momentum or Adam at the same rate — the accumulated velocity rolls through the bump to a deeper minimum. Now crank the rate up on SGD and watch it overshoot and oscillate.

TakeawayMomentum and adaptive methods aren't about which minimum generalises — they change how fast and how reliably you reach a good one. That is why they live in the performance bucket.

Exam · 2026 Q2 — why the learning rate is critical

Mark the true statements:

✓ Too high → unstable training (overshoot / oscillation).
✓ Low → stable and (slowly) converges.
✓ High → moves faster toward the minimum (at the cost of stability).
✓ A low rate lets you fine-tune a pretrained model while keeping its knowledge (≈ 1/10 of the original).
✗ “Must change monotonically / is irreversible” — schedules can warm up, decay, or restart.
✗ “A low rate harms ReLU networks” — it does not.
✗ “Start low and increase for better minima” — standard practice is the opposite (high → low).

06 · Exam intel

What the exam actually tests

Chapter 03 is the most reliably examined of the course — the gen-vs-performance sort appears on all three past papers. Beyond it, two precise facts come up again and again.

Sort a technique: generalization or performance?

Given a list of training tricks, label each. Generalization changes which minimum we prefer (early stopping, weight decay, dropout, augmentation). Performance changes how fast / stably we get there (BatchNorm, momentum, Adam, Xavier/He, ReLU, skip connections, learning rate). Use the interactive sorter above until it is automatic.

Count BatchNorm's parameters

For $C$ channels, BatchNorm has $4C$ parameters: 2C trainable ( $\gamma, \beta$ ) and 2C non-trainable (running $\mu, \sigma^2$ ). The non-trainable running statistics are exactly what populate the “Non-trainable params” line of model.summary().

Dropout & early stopping mechanics

Dropout is on during training, off at inference (the masks are removed and weights scaled). Early stopping halts at the minimum of the validation curve. Both are generalization techniques.

07 · Common mistakes

Where students get this wrong

Filing BatchNorm under "regularization"

BatchNorm primarily speeds and stabilises optimization — a performance technique. Its regularising effect is a mild side-benefit, not its purpose. The same goes for skip connections.

Leaving dropout on at test time

Dropout is off at inference. Keeping it on injects random noise into predictions and throws away the implicit-ensemble averaging that weight scaling provides.

Computing preprocessing stats on the whole dataset

Normalisation means, mean images, and the like must come from the training fold only. Using the full dataset before cross-validation leaks validation/test information into training.

"The learning rate must increase during training"

Standard practice is high → low (decay). The schedule need not be monotonic, and a low learning rate does not specifically harm ReLU networks — in fact a small rate is exactly what fine-tuning wants.

"More capacity always helps"

Training error always falls with capacity, but validation error is U-shaped. Past the sweet spot, extra capacity just memorises noise and generalisation gets worse.

08 · Self-check

Can you answer these?

Four questions in the exact shapes the exam uses. Click an option for instant feedback.

Which of these primarily improves training PERFORMANCE rather than generalization?

A BatchNorm layer over C channels has how many parameters, split how?

What happens to dropout at test (inference) time?

Which statement about the learning rate is TRUE?

09 · Recap

One-screen summary

Chapter 03 — load-bearing ideas

Only generalization (test) error matters; training error is optimistic. Underfit = high bias; overfit = high variance.
Validation selects hyperparameters; the test set is touched once. $K$ -fold is the practical CV; compute preprocessing stats per training fold only.
L2 weight decay = a Gaussian prior on the weights (MAP). Dropout = an implicit ensemble, and is off at inference.
The big sort: generalization (early stopping, weight decay, dropout, augmentation) vs performance (BatchNorm, momentum, Adam, Xavier/He, ReLU, skip connections, learning rate).
BatchNorm is a performance technique with $2C$ trainable + $2C$ non-trainable parameters; the non-trainable running stats appear in model.summary().
Initialization preserves activation variance (Xavier for tanh, He $=2/n_{in}$ for ReLU). The learning rate starts high and decays; the momentum→Adam lineage each fixes the last one’s flaw.