Chapter 09

Famous CNN Architectures

By the end you can read each landmark architecture as a DESIGN LESSON — depth (VGG), parameter collapse (GAP), and efficiency (MobileNet) — rather than memorising specs.

Reading: ~50 min Interactive: 2 widgets Source: Krizhevsky et al. (2012) AlexNet · Simonyan & Zisserman (2014) VGG · Howard et al. (2017) MobileNet

01 · AlexNet and VGG

Depth as strategy

AlexNet (2012) kicked off the deep-learning era of vision (ILSVRC 2012, 15.4% top-5 vs 26.2%). Its lessons were less about the exact shape and more about what made deep training work: ReLU, Dropout, GPU training, and augmentation.

🏗️ AlexNet (2012)

5 conv + 3 FC. 60M params · 94% in the FC layers. Top-5: 15.4%.

🏗️ VGG16 (2014)

13 conv + 3 FC, only 3×3 filters. 138M params (89% FC). Top-5: 7.3%.

VGG’s lesson: replace big kernels with stacks of 3×3. The win compounds with depth:

Numbers 3× Conv(3×3) vs 1× Conv(7×7)

✓ 3× Conv(3×3)

RF = 7×7 · params = 27C · 3 non-linearities.

✗ 1× Conv(7×7)

RF = 7×7 · params = 49C · 1 non-linearity.

key

Where the parameters hide

In both AlexNet and VGG, ~90% of parameters sit in the fully-connected head, not the conv layers (Chapter 07’s LeNet showed the same). That is the problem GAP solves next. (And note LeNet/VGG use “valid”-style padding that shrinks maps — Chapter 07’s 2026 Q7.)

02 · Network in Network & GAP

Collapsing the dense layers

Network in Network (2014) introduced 1×1 “mlpconv” layers and, crucially, Global Average Pooling: average each final feature map to a single number, one per class — no dense layer at all.

Global Average Pooling

F_k = \frac{1}{H\cdot W}\sum_{x=1}^{H}\sum_{y=1}^{W} f_k(x,y)

One number per feature map, and GAP has no learnable parameters.

✓ CNN + GAP

Few params (no big FC) → less overfitting. Shift-tolerant. Each map ≈ one class’s evidence.

✗ CNN + Flatten + Dense

Large FC → many params, higher overfitting risk; not shift-invariant.

key

GAP enables CAM

Because GAP keeps each feature map tied to a spatial location, a GAP→Dense network can produce Class Activation Maps for free (Chapter 10) — localisation from classification labels alone. That is the deeper payoff of removing the dense head.

03 · MobileNet

Depthwise separable convolutions

MobileNet (2017) factorises a standard convolution into two cheap steps, trading a little accuracy for a large efficiency gain:

1️⃣ Depthwise

One 2-D filter per input channel — spatial filtering, no channel mixing. Params: $K\cdot K\cdot C_{in}$ .

2️⃣ Pointwise (1×1)

A 1×1 conv mixing channels — no spatial filtering. Params: $C_{in}\cdot C_{out}$ .

Cost ratio vs standard Conv2D

\frac{\text{depthwise-sep ops}}{\text{standard conv ops}} = \frac{1}{N} + \frac{1}{D_K^2}

For $K=3$ , $N=256$ the ratio $\approx 1/256 + 1/9 \approx 0.115$ — about 8–9× fewer operations. The split of “spatial vs channel” mixing is the same factorisation idea behind the 1×1 bottlenecks of Inception (Chapter 10).

Hands-on 1

Depthwise-separable convolution cost

A standard conv costs D_F²·K²·C_in·C_out. Split it into a depthwise (spatial, per channel) plus a pointwise 1×1 (channel mixing) and the ratio drops to 1/C_out + 1/K². Tune the shapes and watch the saving.

C_in128

C_out256

Output side28

standard conv operations

depthwise-separable (11.5% of standard)

Speedup

8.7×

Op ratio

0.115

Std params

294,912

DS params

33,920

ratio = 1/256 + 1/9 = 0.0039 + 0.1111 = 0.1150

Try thisSet K = 3×3 and C_out = 256: the ratio is 1/256 + 1/9 ≈ 0.115 — about a 9× saving, dominated by the 1/K² term. Bumping C_out higher barely changes it, because 1/C_out is already tiny.

TakeawaySeparating spatial mixing (depthwise) from channel mixing (pointwise 1×1) is the recurring efficiency trick — the same factorisation idea behind Inception's 1×1 bottlenecks.

04 · Visualizing what a CNN learns

Opening the black box

A trained CNN is a stack of millions of numbers. Three classic techniques make those numbers interpretable — and, satisfyingly, they confirm the feature hierarchy from Chapter 01 (edges → parts → objects):

🧱 First-layer filters

Just display the learned kernels as tiny images. They consistently look like oriented edge and colour detectors (Gabor-like) — whatever the dataset. Direct evidence the net learns edges first.

🖼️ Maximally-activating patches

Pick a deep neuron, push many images through, and collect the input patches (within its receptive field) that fire it hardest. Reveals what it detects — faces, text, wheels…

🌈 Gradient ascent

Freeze the weights and optimise the input image (from noise) to maximise a chosen neuron/class score. Synthesises a canonical “what excites this unit” image (the idea behind DeepDream).

Deeper layers can’t be shown directly (their filters span many channels), which is exactly why maximally-activating patches and gradient ascent exist — they probe a neuron’s behaviour rather than its raw weights.

Gradient ascent (feature visualization)

x^* = \arg\max_{x}\; a_{\ell,k}(x) \;-\; \lambda\lVert x\rVert^2

key

Two different "explanation" questions

Gradient ascent answers “what does this neuron detect in general?”. The CAM / Grad-CAM methods of Chapter 10 answer the complementary “where in THIS specific image did the evidence come from?”. Both are needed to trust a model.

05 · The architecture family

Each net pushes one axis

After VGG, progress came from distinct structural ideas. Two get a full treatment in Chapter 10 (ResNet’s skip connections; Inception’s parallel multi-scale branches with 1×1 bottlenecks); the rest of the family rounds out the design space:

➕ ResNet (2015)

Skip connections learn a residual $F(x)+x$ ; an identity gradient highway trains 100+ layers. Full detail in Chapter 10.

🍱 Inception / GoogLeNet

Parallel multi-scale branches + 1×1 bottlenecks; ~5M params. Full detail in Chapter 10.

📏 Wide ResNet

Trade depth for width (more channels per block): fewer layers, similar accuracy, more parallel-friendly.

🔀 ResNeXt

Adds cardinality — many parallel grouped transformations in a block — as a new axis beside depth and width.

🔗 DenseNet

Each layer takes the concatenation of ALL previous feature maps: maximal feature reuse, strong gradient flow, very few parameters.

⚖️ EfficientNet (2019)

Compound scaling: grow depth, width, and input resolution together by a fixed ratio. State-of-the-art accuracy per FLOP.

Hands-on 2

Accuracy vs compute — there is no single best

Each landmark net pushes one axis. The chart plots top-1 accuracy against compute (GFLOPs, log scale); bubble area is the parameter count. Click a bubble to read its design lesson.

Top-1

76%

Compute

4.1 GF

Params

25.6M

ResNet-50: Skip connections train very deep nets; a strong accuracy/compute balance and a common default.

Try thisCompare VGG16 (top-right: accurate but huge and slow) with EfficientNet-B0(top-left: similar accuracy at ~40× fewer FLOPs and ~26× fewer params). The "best" net depends entirely on your compute and latency budget.

TakeawayRead each architecture as a lesson about one trade-off — depth, width, cardinality, connectivity, multi-scale, separability, compound scaling — and pick the one whose trade-off matches your deployment target, not accuracy alone.

key

How to read an architecture-comparison plot

The canonical chart plots top-1 accuracy vs compute (FLOPs), with bubble area = parameter count. The lessons: VGG is accurate but huge and expensive; GoogLeNet/ResNet sit at much better trade-offs; MobileNet/EfficientNet own the efficiency frontier. There is no single “best” network — the right choice depends on your accuracy budget vs your compute/latency/memory budget.

06 · Exam intel

What the exam actually tests

Architectures are tested indirectly — through their design choices in model.summary() and true/false questions (Chapter 07’s 2026 Q7, Chapter 06’s 2026 Q6). Know why each choice exists.

The VGG principle

Stacked 3×3 convs reach the same receptive field as one big kernel with fewer parameters and more non-linearities (three 3×3 = 27C vs one 7×7 = 49C). This is why modern nets favour deep stacks of small filters.

GAP vs flatten + dense

Global Average Pooling averages each map to one number (no parameters), cutting the overfitting-prone FC head and making the net shift-tolerant — and it enables CAM by keeping each map spatially grounded.

Depthwise-separable cost

Depthwise (spatial, per channel) + pointwise 1×1 (channel mixing) costs a fraction $1/N + 1/D_K^2$ of a standard conv — roughly 8–9× cheaper for 3×3. The recurring trick is separating spatial from channel mixing.

07 · Common mistakes

Where students get this wrong

"One 7×7 conv is more efficient than three 3×3"

The 3×3 stack reaches the same receptive field with fewer parameters (27C vs 49C) and adds two extra non-linearities. That is the whole point of VGG.

"Most CNN parameters are in the conv layers"

In AlexNet and VGG ~90% of parameters live in the fully-connected head. The conv stack is parameter-light — which is exactly why GAP (which removes the FC head) helps.

"Global Average Pooling has parameters"

GAP just averages each feature map — zero learnable parameters. Its value is fewer parameters, shift tolerance, and free Class Activation Maps.

"The depthwise step mixes channels"

Depthwise applies one filter per channel with no channel mixing; the pointwise 1×1 does the channel mixing. Splitting these two is what makes it cheap.

"There is one best architecture"

No — it depends on your accuracy-vs-compute budget. VGG is accurate but heavy; MobileNet/EfficientNet own the efficiency frontier; ResNet is a strong balanced default.

08 · Self-check

Can you answer these?

Four questions in the exact shapes the exam uses. Click an option for instant feedback.

Three stacked 3×3 convolutions versus one 7×7 — which statement is true?

Global Average Pooling replaces the dense head by…

A depthwise-separable convolution is cheaper than a standard convolution by roughly…

Which CNN-visualization method synthesises an image (starting from noise) that maximally excites a chosen neuron?

09 · Recap

One-screen summary

Chapter 09 — load-bearing ideas

Read architectures as design lessons: AlexNet made deep training work (ReLU/Dropout); VGG showed deep 3×3 stacks beat big kernels.
~90% of AlexNet/VGG parameters are in the FC head — the motivation for GAP.
GAP averages each feature map to one class score: fewer params, shift-tolerant, and it enables CAM (Chapter 10).
MobileNet = depthwise (spatial, per-channel) + pointwise 1×1 (channel mixing); cost ratio $1/N + 1/D_K^2 \approx$ 8–9× cheaper.
The recurring trick is separating spatial mixing from channel mixing (depthwise-separable, 1×1 bottlenecks).
Visualise a CNN three ways: first-layer filters (Gabor-like edges), maximally-activating patches, or gradient ascent. (CAM/Grad-CAM in Chapter 10 answer “where in THIS image”.)
The post-VGG family each pushes one axis — depth (ResNet), width (Wide ResNet), cardinality (ResNeXt), connectivity (DenseNet), multi-scale (Inception), compound scaling (EfficientNet). Pick by accuracy-vs-compute budget, not accuracy alone.