CNNs — Transfer Learning & Data Scarcity
By the end you can decide — given how much data you have and how similar it is to ImageNet — whether to freeze, fine-tune, or train from scratch, and at what learning rate, plus how augmentation and receptive fields fit in.
01 · Receptive field, revisited
Why stacked 3×3 beats one big kernel
The receptive field (Chapter 07) is the input region a neuron can see. Stacking small filters grows it cheaply: two 3×3 convolutions reach the same 5×5 RF as one 5×5 conv, but with fewer parameters and an extra non-linearity.
🔲 2× Conv(3×3)
RF = 5×5 · ~18C params · 2 non-linearities.
🔲 1× Conv(5×5)
RF = 5×5 · ~25C params · 1 non-linearity.
This is the VGG principle (Chapter 09): prefer deep stacks of 3×3 filters. Re-derive the RF for any stack:
Receptive field calculator
Set each layer's kernel K and stride S. The receptive field grows by (K−1)·jump per layer, and every stride multiplies the jump — so pooling makes the RF balloon. Padding is irrelevant to the RF.
| Layer | K | S | Output | Jump | RF |
|---|---|---|---|---|---|
| Conv 1 | 62 | 1 | 3 | ||
| Pool 2 | 31 | 2 | 4 | ||
| Conv 3 | 29 | 2 | 8 | ||
| Pool 4 | 14 | 4 | 10 |
CNN anatomy — and equivariance ≠ invariance
A CNN bakes in three assumptions: sparse connectivity (local kernels), weight sharing (the same filter everywhere), and a growing receptive field. Weight sharing makes convolution translation-equivariant — shift the input and the feature map shifts the same way. It is pooling (and ultimately global pooling) that converts that equivariance into approximate translation invariance: the prediction stops caring where the feature was. Conflating the two is a classic exam slip.
02 · Data augmentation
Make more data without collecting it
The problem. Deep nets are data-hungry, but labels are scarce. Augmentation manufactures variety with label-preserving transformations — and it is how you build in invariance: to be invariant to a transformation, train on images undergoing it (recall the two-headed horse, Chapter 06).
🔄 Geometric
Flips, crops, rotations, scaling. Label-preserving for most classification.
🎨 Photometric
Colour jitter, brightness/contrast, noise, grayscale.
🔀 Mixup
A convex combination of two images and their one-hot labels.
🧪 TTA
Test-time augmentation: average predictions over augmented copies.
The same mixes the labels — not the dominant one. That symmetry is the whole point of Mixup.
Two augmentation traps
Mixup’s target is the same convex combination of the labels, not the dominant one (2024 Q6). And the transformation must be label-preserving — rotating a clock or an MNIST digit changes its class, and an augmentation applied to only one class becomes a spurious cue.
03 · Transfer learning
Standing on ImageNet’s shoulders
A CNN pre-trained on ImageNet has already learned a general feature hierarchy (edges → textures → parts → objects). Reuse it:
- Load the pre-trained model; remove its classification head.
- Attach a new head for your classes (randomly initialised).
- Feature extraction: freeze the backbone, train only the new head.
- Fine-tuning: unfreeze (some of) the backbone, continue at a much lower LR (~1/10, e.g. 1e-5).
What to do depends on how much data you have and how similar it is to ImageNet:
Freeze, fine-tune, or train from scratch?
The choice is set by two questions: how much data do you have, and how similar is it to the source domain (ImageNet)? Pick both and read off the strategy, the learning rate, and which layers stay frozen.
Freeze the whole backbone and train just a linear head. With little data, fine-tuning the backbone would overfit — the pretrained features already fit a similar domain.
Why the low LR — and do TL before fine-tuning
A high LR makes large updates that overwrite the pre-trained features — catastrophic forgetting. Do transfer learning (head only) before fine-tuning, so the head is sensible before you unfreeze the backbone (2024 Q6). A low LR is precisely what enables fine-tuning without erasing knowledge (2026 Q2).
Exam · 2024 Q6 — transfer learning, Mixup, max-pool
The true statements:
- ✓ VGG transfer recipe: keep only the dense (MLP) head trainable, freeze the conv backbone, train a few epochs.
- ✓ Transfer learning before fine-tuning (given enough data) — warm up the head, then unfreeze.
- ✓ Max-pool backprop routes the gradient only to the argmax location (you must track it).
- ✗ “Mixup’s target is the dominant label” — it is the same convex combination of both labels.
- ✗ “Augmentation layers must be removed at inference” — they are simply inactive, still in the graph.
- ✗ “Any transform is beneficial” — some break label semantics or are unrealistic.
- ✗ “Fine-tune before transfer learning” — the order is reversed.
04 · Measuring performance
Confusion matrix, ROC and AUC
Why accuracy lies. Consider the course’s industrial case study — classifying silicon-wafer defect maps, where genuine defects are rare. A model that predicts “no defect” every time scores 99% accuracy and is completely useless. With imbalanced or asymmetric-cost problems we need richer measures.
Everything starts from the confusion matrix, which tallies predictions against truth for the positive (“defect”) class:
✅ TP / TN
Correctly flagged defects (TP) and correctly cleared good wafers (TN).
⚠️ FP (false alarm)
A good wafer flagged as defective — wasted inspection.
❌ FN (miss)
A real defect passed as good — usually the costly error.
Precision: “of the wafers I flagged, how many were really defective?” Recall: “of the real defects, how many did I catch?”
They trade off as you move the decision threshold — and that trade-off curve is the ROC:
Why accuracy lies — precision, recall, ROC/AUC
Slide the decision threshold and watch the confusion matrix and the ROC point move. Turn on class imbalance (rare positives, like wafer defects): accuracy stays high even for a useless model, while recall and AUC expose it.
| Pred + | Pred − | |
|---|---|---|
| Actual + | 38 | 12 |
| Actual − | 7 | 43 |
ROC curve and AUC
A classifier outputs a score; sweeping the threshold from strict to lenient traces the ROC curve of true-positive rate vs false-positive rate. The Area Under the Curve (AUC) summarises it in one number: AUC = the probability that a random positive is ranked above a random negative. AUC = 1.0 is perfect, 0.5 is random guessing, and — unlike accuracy — it is threshold-independent and robust to class imbalance.
The takeaway for any deployed classifier: pick the metric that matches the cost structure (a missed defect is not a false alarm), and report ROC/AUC rather than a single accuracy number when classes are imbalanced.
05 · Exam intel
What the exam actually tests
Transfer learning and augmentation come as multi-select true/false; metrics come as “why is accuracy the wrong measure here?”.
Transfer learning recipe & order
Freeze the conv backbone and train only the dense head (feature extraction); do this before fine-tuning; fine-tune at a low LR (~1/10) to avoid catastrophic forgetting. Pick freeze vs fine-tune vs scratch by dataset size × similarity to the source.
Augmentation subtleties
Mixup mixes the labels too (same ), not the dominant one. Augmentation layers are inactive at inference, not removed. Not every transform helps — it must be label-preserving.
Metrics under imbalance
Accuracy is misleading when one class is rare. Use precision/recall/F1 at your operating point, and report ROC/AUC — AUC is the probability a random positive outranks a random negative, 0.5 = chance, and it is threshold-independent.
06 · Common mistakes
Where students get this wrong
"Convolution is translation invariant"
Convolution is translation equivariant — shift the input, the feature map shifts. Pooling (and global pooling) is what turns equivariance into approximate invariance. The two are not the same.
"Mixup uses the dominant image's label"
The target is the same convex combination of both one-hot labels, with the same used for the images. Using the dominant label defeats the purpose.
"Augmentation layers must be removed at test time"
They stay in the graph and are simply inactive at inference. And not every transform is safe — label-breaking or unrealistic ones can hurt.
Fine-tuning first, or at a high learning rate
Do transfer learning (frozen backbone, train head) first, then fine-tune at ~1/10 the LR. A high rate overwrites the pretrained features — catastrophic forgetting.
"High accuracy means a good classifier"
Under class imbalance, predicting the majority class alone can score very high accuracy while catching none of the rare positives. Look at recall, F1, and AUC instead.
07 · Self-check
Can you answer these?
Four questions in the exact shapes the exam uses. Click an option for instant feedback.
Why prefer two stacked 3×3 convolutions over one 5×5 convolution?
Mixup forms x̃ = λx_i + (1−λ)x_j. What target ỹ does it use?
When adapting a pretrained CNN to a new task, you should…
On a dataset with 2% positives, a model predicts 'negative' for everything, scoring 98% accuracy. What does AUC reveal?
08 · Recap
One-screen summary
Chapter 08 — load-bearing ideas
- Stacked 3×3 convs reach the same receptive field as a bigger kernel with fewer parameters and more non-linearities (the VGG principle).
- Augmentation manufactures label-preserving variety and builds in invariance; Mixup interpolates images and labels (not just the dominant one).
- Transfer learning: freeze the backbone + train the head (feature extraction), then optionally fine-tune at a much lower LR (~1/10).
- Choose freeze vs fine-tune vs scratch by dataset size × similarity to the source domain.
- A high fine-tuning LR causes catastrophic forgetting; do transfer learning before fine-tuning.
- Under class imbalance accuracy lies; use the confusion matrix (precision/recall/F1) and report ROC/AUC — AUC = P(random positive ranked above random negative), 0.5 = chance, threshold-independent.
- Equivariance ≠ invariance: convolution is translation-equivariant; pooling makes it approximately invariant.