Chapter 07

Convolutional Neural Networks

By the end you can read any CNN's model.summary() — predict every layer's output shape and parameter count — and explain why convolution, not a dense layer, is the right tool for images.

Reading: ~55 min Interactive: 3 widgets Source: Stanford CS231n · LeCun et al. (1998) LeNet-5

01 · Why convolution?

The problem with dense layers on images

The problem. Flatten a modest 32×32×3 image and connect it densely to a 32×32×6 feature map and you need ~4.8 million weights — for one layer. Worse, a dense layer has no notion that pixel (5,5) and (5,6) are neighbours: shuffle the pixels and it learns equally well. Images are highly spatially correlated, and a dense layer throws that structure away.

The fix is to bake two assumptions (inductive biases) into the architecture:

🔲 Sparse connectivity

Each output neuron looks only at a small local patch (the kernel). A useful local feature — an edge, a corner — does not depend on pixels on the far side of the image.

♻️ Weight sharing

The same filter slides over every location. An edge detector useful at $(r,c)$ is useful at $(r',c')$ too. This gives translation equivariance and collapses the parameter count.

why

168 parameters, not 4.8 million

The equivalent Conv2D(6, 3×3) on that 32×32×3 input needs only 168 parameters (6×3×3×3 weights + 6 biases) instead of 4.8M — and it generalises better, because the same feature detector is reused everywhere.

02 · Correlation & conv layers

What a filter actually computes

A correlation is a local, linear operation: the output at $(r,c)$ is a weighted sum of the input pixels in a neighbourhood $U$ , with weights given by the filter $w$ :

Correlation (2D)

T[I](r,c) = \sum_{(u,v)\in U} w(u,v)\, I(r+u,\, c+v)

Hands-on 1

What a convolutional filter computes

Slide a 3×3 filter over the image (a bright top-left block) and read the feature map. Each filter responds to a different structure: edge filters light up along boundaries, blur smooths, sharpen exaggerates. A CNN learns these weights instead of hand-setting them.

input

feature map (Vertical edges)

Try thisCompare Vertical edges and Horizontal edges: each lights up only along the boundary it is tuned for — the vertical filter fires on the left/right edge of the block, the horizontal one on the top/bottom edge.

TakeawayA convolutional layer is a stack of such filters with learnable weights. Early layers learn edge/blob detectors like these; deeper layers compose them into textures, parts, and objects (the hierarchy from Chapter 01).

A convolutional layer is a stack of such filters, but with one crucial extra: it spans all input channels at once. For an input volume with $C$ channels, one filter has shape $h_r\times h_c\times C$ and produces a single 2-D activation map:

Conv layer (one filter, mixes channels)

a(r,c,l) = \sum_{k=1}^{C}\sum_{(u,v)\in U} w_l(u,v,k)\, x(r+u,c+v,k) + b_l

Two consequences students routinely miss, both straight from past exams:

The filter’s depth is not a free choice — it must equal the input depth $C$ , so all channels are processed. Only the spatial size ( $h_r\times h_c$ ) and the number of filters $N_F$ are hyperparameters.
Channels are always summed, so one filter outputs a single 2-D map. $N_F$ filters give $N_F$ output channels. (Contrast image filtering, where R, G, B are filtered separately and stay separate.)

Three exam traps

(1) CNNs actually compute correlation, not convolution — true convolution flips the filter, but since filters are learned the distinction is irrelevant. (2) A Conv2D layer does not require the number of input channels to equal the filter’s spatial size — depth and spatial extent are independent. (3) Stacking conv layers with linear activations collapses to a single conv layer — the non-linearity is what makes depth meaningful.

Deep dive Convolution vs correlation — the filter flip

True convolution is defined with the filter reversed:

T[I](r,c) = \sum_{(u,v)\in U} w(u,v)\, I(r-u,\, c-v) = \sum_{(u,v)\in U} w(-u,-v)\, I(r+u,c+v)

i.e. convolution = correlation with a 180°-flipped filter. The flip makes the maths (associativity, the Fourier convolution theorem) clean in classical image processing. In a CNN the filter weights are learned from scratch, so whether the framework flips them or not just changes which flipped version is learned — the trained network is identical. Keras/PyTorch “Conv2D” layers therefore implement correlation.

03 · CNN arithmetic

Output size and parameter count — memorise these

Given input size $I$ , kernel $K$ , padding $P$ and stride $S$ , the output spatial dimension is:

Output size

O = \left\lfloor \frac{I - K + 2P}{S} \right\rfloor + 1

The number of trainable parameters in a Conv2D layer (the formula behind every model.summary()):

Conv2D parameters

(\underbrace{K \times K \times C_{in}}_{\text{weights per filter}} + \underbrace{1}_{\text{bias}}) \times C_{out}

key

same vs valid padding

“same” padding: set $P = (K-1)/2$ with stride 1 and the output keeps the input size. $K=3 \to P=1$ ; $K=5 \to P=2$ . “valid” padding ( $P=0$ ) shrinks the map by $K-1$ each side.

Worked example (from the slides). A layer with 3 input maps, 2 filters of size 5×5:

Weights: $5 \times 5 \times 3 \times 2 =$ 150
Biases: one per filter = 2
Total = 152 trainable parameters.

Note: a layer with the same hyperparameters can have a different parameter count depending on where it sits in the network, because $C_{in}$ (the input depth) changes.

Exam · 2026 Q7 — conv layers, linearity, LeNet

The true statements:

✓ Conv layers can reduce the spatial extent (stride above 1, or pooling).
✓ A conv layer without non-linearity is a structured sparse linear map (shared weights, Toeplitz-like).
✓ Stacking conv layers with all-linear activations collapses to a single conv layer.
✗ LeNet uses “same” padding — it uses valid (the maps shrink layer by layer).
✗ “Each filter contributes to every output neuron” — each filter produces one output channel.

04 · Pooling & receptive fields

Shrinking space, growing context

Pooling reduces the spatial size of a volume, operating independently on each depth slice. Max-pooling with a 2×2 window and stride 2 keeps the largest value in each window — discarding 75% of the activations and halving $H$ and $W$ . Crucially, pooling has zero learnable parameters.

Trap — pooling has no parameters

Pooling has no parameters and is not a “layer that learns to downsample”. To back-propagate through max-pooling you only need to remember which location held the maximum (the argmax) and route the gradient there.

The receptive field (RF) of a neuron is the region of the original input that can influence its value. RF grows with depth, and pooling/strides accelerate that growth because each stride multiplies the step of every later layer:

Receptive field growth

RF_l = RF_{l-1} + (K_l - 1)\prod_{i=1}^{l-1} s_i

Hands-on 2

Receptive field calculator

Set each layer's kernel K and stride S. The receptive field grows by (K−1)·jump per layer, and every stride multiplies the jump — so pooling makes the RF balloon. Padding is irrelevant to the RF.

Input size64

Layer	Output	Jump	RF
Conv 1	62	1	3
Pool 2	31	2	4
Conv 3	29	2	8
Pool 4	14	4	10

Final RF

Final output

Covers input?

Try thisReproduce the exam stack: Conv k3 → Pool k4 s4 → Conv k5 → Pool k3 s3 → Conv k7 → Pool k2 s2 → Conv k9, input 500. The RF walks 3 → 6 → 22 → 30 → 102 → 114 → 306. The big jumps come right after each pool.

TakeawayReceptive field grows slowly through convs but leaps after each stride/pool, because the stride multiplies the step of every later layer. That is how a deep stack "sees" the whole image without a giant kernel.

Exam · 2025 Q7 — compute the receptive field

Conv(k3) → MaxPool(4) → Conv(k5) → MaxPool(3) → Conv(k7) → MaxPool(2) → Conv(k9) → MaxPool(1), input 500×500. Walk it front-to-back: 3 → 6 (jump 4) → 22 → 30 (jump 12) → 102 → 114 (jump 24) → 306. Padding does not affect the RF (only the output size), and the RF is not “the whole image whatever the input”.

05 · LeNet-5 & model.summary()

The first successful CNN (LeCun, 1998)

LeNet-5 recognised handwritten digits (MNIST). It interleaves Conv + activation + pooling to extract features, then hands a flattened vector to a small MLP for classification — the template every later CNN follows.

from keras.models import Sequential
from keras.layers import Dense, Flatten, Conv2D, AveragePooling2D

model = Sequential()
model.add(Conv2D(6,  (5,5), activation='tanh', input_shape=(32,32,1), padding='valid'))
model.add(AveragePooling2D(pool_size=(2,2)))
model.add(Conv2D(16, (5,5), activation='tanh', padding='valid'))
model.add(AveragePooling2D(pool_size=(2,2)))
model.add(Flatten())                       # 5×5×16 = 400
model.add(Dense(120, activation='relu'))
model.add(Dense(84,  activation='relu'))
model.add(Dense(10,  activation='softmax'))
model.summary()

Running model.summary() prints the per-layer shapes and parameter counts. Reproduce it by hand:

🔢 Conv1 — 156

6 × (5×5×1 + 1)

🔢 Conv2 — 2,416

16 × (5×5×6 + 1)

🔢 Dense 120 — 48,120

400×120 + 120

🔢 Dense 84 — 10,164

120×84 + 84

🔢 Dense 10 — 850

84×10 + 10

📊 Total — 61,706

conv 2,572 + MLP 59,134

key

Where the parameters live

The two conv layers hold just 2,572 params; the MLP holds 59,134 — ~96%. And an RGB input would only change the first conv (156 → 456, since $C_{in}$ : 1 → 3); a dense net taking the whole image would triple its entire first layer. That is the parameter-efficiency of convolution in one comparison.

Build any architecture and verify the shapes and parameters live:

Hands-on 3

model.summary() calculator

Edit the hyperparameters and read the per-layer output shapes and parameter counts — exactly the table the exam asks you to fill in. Conv params are (K·K·C_in + 1)·C_out; pooling and flatten add none. Watch where the parameters actually live.

Input32

Channels1

Kernel5

Conv1 filters6

Conv2 filters16

Dense units120

Layer	Output shape	Params
Input	32×32×1	0
Conv2D(6, 5×5)	28×28×6	156
MaxPool 2×2	14×14×6	0
Conv2D(16, 5×5)	10×10×16	2,416
MaxPool 2×2	5×5×16	0
Flatten	400	0
Dense(120)	120	48,120
Dense(10)	10	1,210

Total params

51,902

Conv params

2,572

In dense head

95%

Try thisSet input 32, 1 channel, kernel 5, conv1 = 6, conv2 = 16, dense = 120 — that's LeNet-5: Conv1 = 156, Conv2 = 2,416, Dense = 48,120, total 61,706. Now flip channels 1 → 3: only Conv1 changes (156 → 456). That is the parameter-efficiency of convolution.

TakeawayConv params depend on C_in — which changes with depth — so the same layer can cost differently in different positions. The convs stay light; the flattened dense head usually holds the overwhelming majority of the parameters.

Skip Add vs U-Net Concat

A residual Add needs the two branches to have equal channels (hence the 1×1 convs in ResNet blocks). A U-Net skip concatenates, which raises the channel count — so the next conv sees more input channels and has more parameters. Mixing these up is the most common model.summary() mistake (2026 Q10).

06 · Latent representations & retrieval

The feature space is the real payoff

Recall Chapter 06’s cliffhanger: a Nearest-Neighbour classifier failed on raw pixels because L2 pixel distance is not perceptual distance. A trained CNN solves exactly that. The activations of the penultimate layer (just before the classifier) form a learned embedding — a vector that summarises the image’s content — and in that space Euclidean distance finally matches semantic similarity:

A CNN as feature extractor + classifier

\underbrace{\phi(\mathbf{x})}_{\text{conv stack} \to \text{ embedding}} \in \mathbb{R}^d \;\xrightarrow{\;\text{linear}\;}\; \text{class scores}

🌌 t-SNE shows structure

Project $\phi(\mathbf{x})$ to 2-D with t-SNE and the classes form clean clusters — the very clustering that was absent in pixel space.

🔎 Image retrieval

Embed a query image, return its nearest neighbours in feature space — they are visually/semantically similar, the basis of content-based image search.

🎯 1-NN now works

The Nearest-Neighbour classifier that failed on pixels succeeds in latent space: nearby embeddings share a class.

key

This is why transfer learning works

Because the embedding captures general visual structure (not just the training classes), it can be reused for new tasks — freeze the conv stack, retrain only the small head. That is the entire premise of Chapter 08. The CNN’s lasting value is the representation $\phi$ , not the final classifier bolted on top of it.

07 · Exam intel

What the exam actually tests

Filling in a model.summary() is the single most reliable point-earner of the whole course — it appears once per paper.

Fill in model.summary() for a multi-branch net

Per layer: output size $O = \lfloor(I-K+2P)/S\rfloor + 1$ ; Conv2D params $(K^2 C_{in}+1)\,C_{out}$ ; pooling and flatten have zero params; a Dense layer is $\text{in}\times\text{out} + \text{out}$ . Track $C_{in}$ carefully — it changes with depth — and remember BatchNorm adds $2C$ trainable + $2C$ non-trainable.

Receptive field arithmetic

Walk front-to-back: $RF_l = RF_{l-1} + (K_l-1)\cdot\text{jump}$ , and multiply the jump by each stride. Padding never changes the RF. The big jumps come right after each pool.

Residual Add vs concatenation

A residual Add needs equal channels on both branches (use a 1×1 conv to match). A U-Net concat raises the channel count, so the following conv has more input channels and more parameters. This is the most common summary-table slip.

08 · Common mistakes

Where students get this wrong

"The filter depth is a hyperparameter"

A filter’s depth must equal the input depth $C$ — it is fixed, not chosen. Only the spatial size and the number of filters are hyperparameters.

"Channels stay separate like RGB filtering"

Inside a conv layer the channels are summed, so one filter outputs a single 2-D map. $N_F$ filters give $N_F$ output channels. Image filtering keeps R/G/B separate; a conv layer does not.

"Pooling has parameters / learns to downsample"

Max/average pooling has zero learnable parameters. Backprop through max-pool just routes the gradient to the argmax location.

"Stacking conv layers always adds power"

Without non-linearities between them, stacked convs collapse to a single conv. Depth only helps with a non-linear activation after each layer.

"Padding changes the receptive field"

Padding changes the output size, not the receptive field. The RF depends only on kernels and strides.

09 · Self-check

Can you answer these?

Four questions in the exact shapes the exam uses. Click an option for instant feedback.

A Conv2D layer with 16 filters of size 5×5 on an input with 6 channels has how many trainable parameters?

Input 28×28, kernel 5×5, valid padding (P=0), stride 1. What is the output spatial size?

How many learnable parameters does a 2×2 max-pooling layer have?

A single convolutional filter applied to a volume with C input channels produces…

10 · Recap

One-screen summary

Chapter 07 — load-bearing ideas

Convolution = local + weight-shared correlation; it encodes that image features are local and translation-equivariant, which dense layers ignore.
One filter spans all input channels and outputs a single 2-D map; $N_F$ filters → $N_F$ output channels. Filter depth = input depth (not a hyperparameter).
Output size $O = \lfloor(I-K+2P)/S\rfloor + 1$ ; Conv2D params $(K\cdot K\cdot C_{in} + 1)\cdot C_{out}$ . Pooling has zero parameters.
CNNs compute correlation, not convolution — but with learned filters it makes no difference.
In LeNet ~96% of parameters live in the MLP; conv layers are parameter-light, and RGB only inflates the first conv.
The penultimate layer is a learned embedding where Euclidean distance ≈ semantic similarity: t-SNE clusters by class, 1-NN and image retrieval work — and that representation is why transfer learning works (Chapter 08).