Chapter 10

WSL, Explanations & Advanced Architectures

By the end you can read BatchNorm's parameters off a model.summary, explain how a classifier localises objects it was never taught to localise (CAM), and reason about why residual skips ADD while U-Net skips concatenate.

Reading: ~55 min Interactive: 2 widgets Source: Ioffe & Szegedy (2015) Batch Normalization · Zhou et al. (2016) CAM · Selvaraju et al. (2017) Grad-CAM · He et al. (2015) ResNet · Szegedy et al. (2014) GoogLeNet

01 · Preprocessing and BatchNorm

Normalising the data, and the activations

Before a single weight updates, the input data should be conditioned. Un-normalised inputs make the loss surface elongated, so gradient descent zig-zags; normalisation makes it more isotropic and trainable. The classic toolkit:

📍 Mean subtraction

Subtract the (per-channel or per-pixel) mean image so features are zero-centred. The most common step.

📐 Normalization

Divide by the standard deviation so every dimension has comparable scale (unit variance).

🌀 PCA whitening

Rotate onto the principal axes and scale each to unit variance — decorrelates the features. Powerful but rarely used on raw images (expensive, amplifies noise).

Compute statistics on training data only

The mean/std (and PCA basis) must come from the training set and then be applied unchanged to validation/test — computing them over all data leaks information (Chapter 03’s data-leakage trap, revisited in 2026 Q7).

Preprocessing fixes the input distribution once. Batch Normalization (Ioffe & Szegedy, 2015) extends the idea inside the network: it normalises each channel over the mini-batch, then restores flexibility with a learnable scale $\gamma$ and shift $\beta$ :

Batch Normalization

y_{i,j} = \gamma_j\,\frac{x_{i,j} - \mu_j}{\sqrt{\sigma_j^2 + \epsilon}} + \beta_j

$\mu_j,\sigma_j$ are computed per channel, per mini-batch at training time; at inference, running averages replace them and BN fuses into the adjacent conv as a fixed linear map.

The parameter count per BN layer with $C$ channels is 4C: 2C trainable ( $\gamma,\beta$ ) plus 2C non-trainable (running $\mu,\sigma^2$ ).

exam

Two things the exam checks

BatchNorm is a PERFORMANCE technique (faster convergence, higher learning rate, mild regularisation) — not primarily a generalization technique (Chapter 03’s big sort). And its 2C non-trainable running stats are exactly what populates the “Non-trainable params” line of a model.summary() (Chapter 07).

02 · Multi-label classification

When the classes aren’t mutually exclusive

Ordinary (multi-class) classification assumes the labels compete: exactly one is correct, so we use softmax + categorical cross-entropy and the outputs sum to 1. But a single street photo can contain a person and a dog and a car at once — the labels are not mutually exclusive. Forcing softmax here is wrong: raising one class’s probability would suppress the others that are also present.

The fix is to treat each class as an independent yes/no question:

🎲 Multi-class (mutually exclusive)

Softmax over K classes + categorical cross-entropy. Outputs sum to 1; pick the argmax. “Which one?”

🏷️ Multi-label (co-occurring)

One sigmoid per class + (summed) binary cross-entropy. Outputs are independent; threshold each. “Which ones?”

Multi-label loss (independent Bernoulli per class)

\mathcal{L} = -\sum_{c=1}^{C}\big[y_c\log\sigma(s_c) + (1-y_c)\log(1-\sigma(s_c))\big]

Each output passes through its own sigmoid, so any number of labels (zero, one, or many) can fire above threshold. This is exactly the Bernoulli-vs-Categorical noise-model choice from Chapter 02 — applied per label instead of once over the whole vector.

Hands-on 1

Softmax vs. sigmoid — one label or many?

The same four logits feed two output heads. Softmax couples the classes (probabilities sum to 1 → exactly one winner). Sigmoid scores each class on its own (any subset above 0.5 fires). Raise both Person and Dog and watch softmax pick only one while sigmoid lets both through.

Person2.0

Dog1.6

Car-1.0

Tree0.2

Class	Softmax	Sigmoid	Fires?
Person	53%	88%	✓
Dog	36%	83%	✓
Car	3%	27%	—
Tree	9%	55%	✓

Softmax says

Person

Σ softmax

100%

Sigmoid fires

3 classes

Softmax is forced to choose Person alone, but the image plausibly contains Person + Dog + Tree. Multi-label (sigmoid) is the honest head here.

Try thisPush Person and Dog both to +2. Softmax splits the mass (~50/50, one winner), but sigmoid puts both well above 0.5 — exactly what you want when a photo holds a person and a dog.

TakeawaySingle-label classification uses softmax + cross-entropy; multi-label uses per-class sigmoid + binary cross-entropy summed over classes. Pick the head from the label structure, not the backbone.

key

The head follows the label structure, not the backbone

You can bolt either head onto the same convolutional backbone. What forces the choice is whether the labels co-occur. Co-occurring → per-class sigmoid + binary cross-entropy; mutually exclusive → softmax + categorical cross-entropy.

03 · Explaining predictions

Localising with only classification labels

The idea. A network trained only to classify can still say where the object is — weakly supervised localisation. Class Activation Maps need a specific architecture: a GAP layer followed by a single FC layer. The map for class $c$ re-weights the feature maps by that class’s FC weights:

Class Activation Map

M_c(x,y) = \sum_k w_k^c\, f_k(x,y)

Thresholding $M_c$ (e.g. at $0.2\cdot\max$ ) gives a class-specific bounding box — the same image highlights different regions for different classes.

This is why Global Average Pooling matters (Chapter 09): GAP keeps each feature map tied to a spatial location, so the classifier’s own weights double as a spatial explanation. The only labels were image-level classes, yet the network reveals object position.

Hands-on 2

Class Activation Mapping — a weighted sum of feature maps

The final conv block emits feature maps f_k. For class c the heatmap is M_c = Σ_k w_k^c · f_k — the classifier's own weights, reused as a spatial explanation. Pick a class and watch which maps it amplifies.

f1 · fur / ears

w = +1.0

f2 · wheel / window

w = +0.2

f3 · headlight

w = -0.4

M(Cat)

Cat leans on the left "fur/ears" map, suppresses the car cues. The bright region is the model's evidence for Cat — note it shifts across the image with no bounding-box labels.

Try thisToggle Cat ↔ Car. The feature maps never change — only the class weights do — yet the hot region jumps from left to right. That is the whole idea: localisation falls out of the classifier weights.

TakeawayCAM needs a GAP layer feeding a single linear classifier. Grad-CAM removes that constraint by using gradients ∂y_c/∂f_k as the weights, so it works on any CNN — but the recipe "weight the feature maps, sum, ReLU, upsample" is identical.

There is a small family of such explanation methods, increasing in generality:

✨ Saliency map

Gradient of the class score w.r.t. the input pixels: which pixels, if nudged, most change the prediction. A raw, noisy per-pixel importance map.

🗺️ CAM

Requires GAP → Dense. Class-specific and clean — but limited to that architecture.

🎯 Grad-CAM

Works on any CNN: weights conv feature maps by the gradient of the class score. Generalises CAM.

Augmented Grad-CAM goes further: it fuses Grad-CAM maps from many augmented views of the image and uses super-resolution to produce a high-resolution, sharper heatmap than a single low-res feature map allows.

why

Research application: illegal landfill detection (Perivallon / ODIN)

These maps aren’t just pretty pictures. In the lab’s remote-sensing project, a CNN trained on image-level “landfill / no landfill” labels uses CAM-style maps to point inspectors at the suspected location in satellite imagery — localisation learned without a single bounding-box annotation. That is the practical promise of weakly supervised learning.

04 · ResNet & residual learning

An additive highway for gradients

The problem. Past ~20 layers, plain deep nets degrade — even the training error rises. ResNet (He et al., 2015) fixes this by learning a residual instead of the full mapping:

Residual block

\mathbf{y} = \mathcal{F}(\mathbf{x}) + \mathbf{x}, \qquad \mathcal{F}(\mathbf{x}) = \mathcal{H}(\mathbf{x}) - \mathbf{x}

The identity skip creates a gradient highway: $\partial L/\partial x = \partial L/\partial y\cdot(\partial \mathcal{F}/\partial x + I)$ , and the +I always passes the gradient through, undiluted by weight products. That trains 100+ layer networks (ILSVRC 2015, 3.57% top-5).

key

Add (ResNet) needs equal channels — Concat (U-Net) does not

A residual Add requires the skip and main branches to have the same shape — which is exactly why ResNet blocks use a 1×1 conv to match channels before adding. Contrast U-Net, whose skip concatenates and raises the channel count (Chapter 11). And the additive highway is the same trick as the LSTM cell-state update (Chapter 04).

This shape-consistency reasoning is the crux of three past code/architecture questions — what can be plugged between activation maps, and how a residual net’s model.summary() works:

2024 · Which layers map 128×128×32 → 128×128×64?

Only snippets that keep H,W = 128 and set C = 64 fit: a 1×1 Conv2D(64), a 3×3 Conv2D(64, 'same') (optionally followed by a stride-1 ‘same’ pool), or two ‘same’ convs ending in 64. GAP collapses the spatial size, UpSampling doubles it, an Activation or a 32-filter conv keep 32 channels, and an Add([x1, x2]) with x1 at 32 and x2 at 64 channels fails — a residual Add needs matching shapes. The target shape is fully specified, so “we can’t tell” is wrong.

2024 / 2025 · Fill the model.summary() of a residual regressor

Both papers hand you a residual-regressor listing and ask you to complete model.summary(). The mechanics are Chapter 07’s — Conv params $=(k_h\cdot k_w\cdot C_{in}+1)\cdot C_{out}$ , 'same' keeps $H,W$ , pooling divides them, GAP → $1{\times}1{\times}C$ with 0 params, Dense $=$ units $\cdot$ (inputs+1) — plus one new row: BatchNorm adds 4C params, half of them non-trainable. The Add rows carry 0 params but are why the two branches must share a channel count.

Skip connections do not block fine-tuning

Skip connections do not “prevent weight updates” or make a backbone (e.g. ResNet50) a poor fine-tuning candidate (2026 Q7) — they help gradients flow, so ResNets fine-tune extremely well. They are also not a parameter-saving trick.

05 · Inception (GoogLeNet)

Multi-scale in parallel, cheaply

Inception v1 (GoogLeNet, 2014) runs parallel branches with different kernel sizes (1×1, 3×3, 5×5, max-pool) and concatenates them, capturing features at multiple scales in one module. The trick that makes it affordable is the 1×1 bottleneck convolution, which reduces depth before the expensive 3×3/5×5 convolutions — cutting operations from ~854M to ~358M, for only ~5M parameters (vs VGG’s 138M). It won ILSVRC 2014 at 6.7% top-5.

key

The 1×1 convolution is the quiet workhorse of modern CNNs: it mixes channels and changes depth without touching spatial size — used here for bottlenecks, in ResNet to match Add channels, and in Chapter 11 to convolutionalize dense layers.

06 · Exam intel

What the exam actually tests

This chapter is exam-heavy: BatchNorm parameter counts and ResNet’s additive skip show up in nearly every model.summary() and true/false set. Know the numbers and the shape rules cold.

BatchNorm = 4C params, half non-trainable

A BatchNorm layer over $C$ channels has 4C parameters: 2C trainable ( $\gamma,\beta$ ) and 2C non-trainable (running $\mu,\sigma^2$ ). Those non-trainable stats are the source of the “Non-trainable params” line in model.summary(). BatchNorm is a performance technique, not a generalization one.

Softmax vs sigmoid heads

Mutually-exclusive labels → softmax + categorical cross-entropy (outputs sum to 1). Co-occurring labels → one sigmoid per class + summed binary cross-entropy, each thresholded independently. The backbone is identical; the label structure picks the head.

CAM needs GAP → Dense; Grad-CAM needs neither

$M_c = \sum_k w_k^c f_k$ reuses the FC weights of a GAP-headed net, so it is architecture-specific. Grad-CAM swaps those weights for gradients $\partial y_c/\partial f_k$ and therefore works on any CNN. Both give weakly-supervised localisation from image-level labels.

Residual Add vs U-Net Concat

A residual Add needs matching shapes (hence the 1×1 conv to fix channels); a U-Net skip concatenates and grows the channel count. The +I term is the gradient highway that lets ResNet train 100+ layers — it does not impede fine-tuning.

07 · Common mistakes

Where students get this wrong

"All BatchNorm parameters are trainable"

Only 2C of a BatchNorm layer’s 4C params are trainable ( $\gamma,\beta$ ). The other 2C (running $\mu,\sigma^2$ ) are non-trainable and feed the “Non-trainable params” total in model.summary().

"Use softmax for multi-label classification"

Softmax couples the classes (they sum to 1), so it suppresses co-occurring labels. Multi-label needs one sigmoid per class with summed binary cross-entropy, thresholded independently.

"Compute preprocessing statistics on the whole dataset"

Mean/std and the PCA basis must come from the training set only. Using the full dataset (including validation/test) before cross-validation leaks information — 2026 Q7’s trap.

"CAM works on any CNN"

Plain CAM requires a GAP → single Dense head. The method that works on any architecture is Grad-CAM, which uses gradients in place of the FC weights.

"Residual skips block fine-tuning / save parameters"

The identity skip helps gradient flow (ResNets fine-tune well) and adds no parameters of its own. Its job is the gradient highway, not parameter saving — and its Add needs equal channels.

08 · Self-check

Can you answer these?

Four questions in the exact shapes the exam uses. Click an option for instant feedback.

A BatchNormalization layer over 64 channels reports how many parameters in model.summary(), and how many are non-trainable?

An image can contain a person, a dog, and a car simultaneously. Which output head and loss fit this task?

You have a generic CNN with several dense layers (no GAP head) and want a class-specific localisation heatmap. Which method applies?

In a residual block y = F(x) + x, why do ResNet blocks often insert a 1×1 convolution on the skip path?

09 · Recap

One-screen summary

Chapter 10 — load-bearing ideas

Preprocess inputs (mean subtraction, normalization, PCA whitening) using training statistics only; BatchNorm extends the same idea inside the net.
BatchNorm is a PERFORMANCE technique with 4C params — 2C trainable ( $\gamma,\beta$ ) + 2C non-trainable (running stats); the non-trainable half appears in model.summary().
Multi-class (mutually exclusive) = softmax + categorical CE; multi-label (co-occurring) = one sigmoid per class + summed binary CE, thresholded independently.
Explanation family: saliency (gradient w.r.t. input pixels), CAM (needs GAP→Dense), Grad-CAM (any CNN via gradients), Augmented Grad-CAM (super-res). CAM-style maps give weakly-supervised localization (e.g. landfill detection from image labels).
ResNet learns a residual $\mathcal{F}(x)+x$ ; the $+I$ identity skip is a gradient highway (same idea as the LSTM cell state) that enables 100+ layers.
ResNet skips ADD (need equal channels → 1×1 conv to match); U-Net skips CONCATENATE (raise channels). Skips help fine-tuning, they don’t block it.
Inception uses parallel multi-scale branches with 1×1 bottlenecks (~5M params). The 1×1 conv mixes channels / changes depth without touching space.