Chapter 06

Image Classification

By the end you can explain why a linear classifier on raw pixels is literally template matching by correlation, why that's too weak, and why pixel distance fails — the motivation that makes CNNs necessary.

Reading: ~45 min Interactive: 1 widgets Source: Stanford CS231n · Goodfellow, Bengio & Courville Ch. 9

01 · The CV landscape & pixels

From recognition tasks to pixels as a tensor

Computer vision is a family of recognition problems of increasing structure — and the next seven chapters tour through them. Worth seeing the whole map first, because they share foundations:

🏷️ Classification

One label for the whole image. This chapter → Chapter 09.

📦 Detection

Labels + bounding boxes for many objects. Chapter 12.

🎨 Segmentation

A label for every pixel. Chapter 11.

🤸 Pose / captioning / inspection

Keypoints, descriptions, defect maps — the same backbone, different heads.

Whatever the task, it starts from the same raw object. A digital image is a 3-D tensor of pixel intensities in $[0,255]$ ; a video adds a time axis:

Image / video tensor

I \in \mathbb{R}^{R\times C\times 3}, \qquad V \in \mathbb{R}^{R\times C\times 3\times T}

The dimensionality explodes fast — a full-HD frame is ~6 MB uncompressed.

For classification the task is a function $K: I \to \Lambda$ assigning a label from a fixed category set. The question is how to build $K$ — and the naive answer (flatten and use a dense net) is where the trouble starts.

02 · Correlation & template matching

Sliding a filter is matching a template

Correlation is a local linear operation: slide a filter $w$ over the image and take a weighted sum at each location. The filter alone defines the operation:

2D correlation

(I \otimes w)(r,c) = \sum_{(u,v)\in U} w(u,v)\, I(r+u,\, c+v)

why

Use the thing you seek as the filter

Use a small picture of the thing you’re looking for as the filter. Where the patch under the filter looks like the filter, the weighted sum peaks. Correlation is the most rudimentary template matching — and it is translation invariant (the same response wherever the pattern appears).

One catch: raw correlation also fires on bright flat regions (a white patch correlates highly with anything), so it must be normalised to work as template matching. For RGB the correlation is computed channel-wise and summed.

Hands-on 1

Correlation is template matching

Slide a 3×3 template across the image and read the correlation map (a weighted sum at every position). Move the pattern with the sliders — the peak (green) follows it, which is what translation invariance means. Swap the template and the response changes with how much the patch resembles the filter.

Pattern row5

Pattern col5

image (12×12)

template

correlation map

Peak value

Peak cell

(4, 4)

Template

Cross

Try thisKeep the Cross template and drag the pattern around — the green peak tracks it everywhere (translation invariance). Now switch to Vertical bar: the peak value drops, because the cross only partially matches a bar filter.

TakeawayA linear image classifier learns one template (weight image) per class; the class score is the correlation of the image with that template, and the prediction is the maximum correlation — not the Euclidean-nearest template.

Exam · 2026 Q6 — correlation & convolution fundamentals

The true statements in the big multi-select:

✓ A CNN’s convolutional backbone is a learnable non-linear filter scanning the whole image.
✓ Convolution and correlation are translation-invariant (ignoring boundaries/padding) — weight sharing makes them so.
✓ A trained linear classifier ≡ per-class correlation templates; the largest correlation (± bias) is the predicted class.
✓ Correlation is a rudimentary template matcher (slide a template, score by correlation).
✗ “Convolution is a cascade of local linear ops each followed by a ReLU” — it is a single local linear operation.

03 · The linear classifier as templates

What a one-layer network actually learns

Flatten the image to $\mathbf{x} \in \mathbb{R}^d$ (CIFAR-10: $d = 32\times32\times3 = 3072$ ) and a one-layer network computes class scores:

Linear classifier

\mathbf{s} = W\mathbf{x} + \mathbf{b}, \quad s_i = W[i,:]\cdot\mathbf{x} + b_i, \quad \hat{y} = \arg\max_i s_i

Each row $W[i,:]$ , reshaped to image size, is a learned template, and the score is the correlation of the image with that template at the centre pixel. So a linear classifier = one template per class, and the decision is max correlation (up to bias) — not nearest template by Euclidean distance.

key

Softmax: needed for the loss, not the prediction

For the prediction you can drop softmax — it only normalises the scores and doesn’t change the argmax. But to train with cross-entropy you do need a sigmoid/softmax output, otherwise the scores aren’t probabilities. This subtlety is a favourite exam point.

Training it: two losses on the same scores

The scores $\mathbf{s} = W\mathbf{x} + \mathbf{b}$ are turned into a learning signal by one of two classic losses:

📏 Hinge (multiclass SVM)

Penalise any wrong class whose score comes within a margin $\Delta$ of the correct one — and nothing once the margin is met. Cares only about the boundary.

📊 Cross-entropy (softmax)

Turn scores into probabilities and maximise the correct class’s log-probability. Always pushes for more confidence, even when already correct.

Hinge vs cross-entropy (per example)

L_{\text{hinge}} = \sum_{j\neq y}\max(0,\, s_j - s_y + \Delta), \qquad L_{\text{CE}} = -\log\frac{e^{s_y}}{\sum_j e^{s_j}}

Both are paired with a regularisation term (typically L2, $\gamma\sum W^2$ ) so the weights stay small and the templates generalise — exactly the MAP view from Chapter 03. In modern CNNs cross-entropy dominates, but the hinge loss is the conceptual link to the SVM era that preceded deep learning.

What does it learn? On CIFAR-10 the “horse” template famously has two heads — the training set has horses facing both ways, and one linear template can only average them. That single picture explains both the weakness of linear classifiers and the principle of data augmentation: to get invariance to a transformation, train on images undergoing it.

Exam · 2025 Q6 — a no-hidden-layer MLP on images

For input → flatten → output (no hidden layer), the true statements:

✓ It learns one template (weight image) per class and scores by correlation $w^\top x$ .
✓ It needs a softmax/sigmoid head if trained with cross-entropy.
✗ “Assigns the image to the Euclidean-nearest template” — it is the largest correlation, not distance.
✗ “Not convolutional, so translation augmentation is pointless” — a dense classifier is not translation invariant, so augmentation still helps.
✗ “The predicted class might not be the largest score” — softmax is monotonic, so the argmax is unchanged.

04 · Why images are hard

The case for a special architecture

A linear template can’t cope with the real challenges: high dimensionality, label ambiguity, transformations (illumination, deformation, viewpoint), large inter-class variability, and — the killer — that perceptual similarity is not pixel similarity.

Why not just a deep MLP? Two reasons:

Dimensionality. A single hidden layer of 1536 units on CIFAR-10 already needs ~4.7M parameters — and it still treats pixels as an unordered vector.
Linearity collapses. Without non-linearities, stacked linear layers reduce to one: $W_3W_2(W_1x+b_1)+\dots = \tilde W x + \tilde b$ . Depth only helps with non-linear activations.

And the lazy alternative: Nearest Neighbours

If a single template per class is too weak, why not keep all the training images and classify by similarity? That is the Nearest Neighbour classifier — the simplest non-parametric baseline:

1-NN: assign the label of the single closest training image (by, say, L2 pixel distance).
K-NN: take a majority vote of the $K$ closest. Larger $K$ smooths the boundary and resists label noise, at the cost of blurring fine class detail.

It has no training (just store the data) but is expensive at test time (compare against every stored image), and — crucially — its accuracy is at the mercy of the chosen distance metric.

k-NN doesn't save you either

A k-NN classifier on raw pixels fails because L2 pixel distance is not perceptual distance: a small shift, a brightness change, or an occlusion moves an image far in pixel space while leaving its class obvious to a human. t-SNE on CIFAR-10 shows no class clustering in pixel space — but clear clustering in CNN feature space. The fix isn’t a cleverer classifier, it’s a better representation to measure distance in (Chapter 07).

The fix is an architecture that builds in locality and translation equivariance and learns non-linear, hierarchical features — the Convolutional Neural Network of Chapter 07, whose backbone is exactly a “learnable non-linear filter” generalising the correlation we started with.

05 · Exam intel

What the exam actually tests

The recurring points are all variations on “a linear classifier is template matching by correlation”.

Linear classifier = templates

One learned template per class; the decision is the largest correlation ( $w^\top x$ , ± bias), not the Euclidean-nearest template. You need a softmax/sigmoid only to train with cross-entropy — never to take the argmax. The weight image has a fixed size $H\cdot W\cdot C$ , so a different channel count is incompatible.

Correlation & convolution properties

Convolution/correlation is a single local linear operation (no built-in ReLU cascade), it is translation-invariant thanks to weight sharing, and a CNN backbone is a learnable non-linear filter scanning the whole image.

Match the model to the signal (2025 Q2)

For a multi-sensor 1-D signal (taps/knocks on a surface), a windowed 1-D CNN (one channel per sensor) can both count (temporal structure) and localise (which sensor). A memory-less per-instant feedforward net can localise but cannot count over time.

06 · Common mistakes

Where students get this wrong

"The linear classifier picks the Euclidean-nearest template"

It picks the maximum correlation ( $w^\top x$ , up to bias), not the smallest distance. Correlation and distance order images differently — this is a classic exam trap.

"You need softmax to make a prediction"

Softmax is monotonic, so it never changes the argmax. You need it only to turn scores into probabilities for cross-entropy training, not for inference.

"A dense classifier is translation invariant"

A flattened-input dense net is not translation invariant — shifting the object changes which weights it lands on. That is exactly why translation augmentation still helps such a model.

"k-NN on pixels works with the right k"

The problem isn’t $k$ — it’s the metric. L2 pixel distance is not perceptual distance, so no $k$ rescues it. You need a learned feature space (CNN) to measure similarity in.

"Stacking linear layers adds power"

Without non-linearities, $W_3W_2W_1 = \tilde{W}$ — the whole stack collapses to a single linear map. Depth only buys expressiveness through non-linear activations.

07 · Self-check

Can you answer these?

Four questions in the exact shapes the exam uses. Click an option for instant feedback.

A one-layer linear image classifier assigns an image to a class by…

Do you need a softmax to make a prediction with a linear classifier?

Why does a k-NN classifier on raw pixels fail for images?

Sliding-window correlation (template matching) is translation invariant because…

08 · Recap

One-screen summary

Chapter 06 — load-bearing ideas

An image is an $R\times C\times 3$ tensor; flattening it for an MLP throws away spatial structure and explodes the parameter count.
Correlation = sliding a filter = rudimentary, translation-invariant template matching (normalise it to use as matching).
A linear classifier learns one template per class; the score is correlation with that template, and the decision is max correlation — not Euclidean nearest.
Softmax/sigmoid is needed to compute cross-entropy, but not to take the argmax prediction.
Pixel distance is not perceptual distance, and stacked linear layers collapse — so images need CNNs (Chapter 07), not MLPs or k-NN.