Chapter 11

Semantic Segmentation

By the end you can explain how a classifier becomes a dense predictor (FC → 1×1 conv), how transpose convolution learns to upsample, and exactly why U-Net concatenates its skips — the facts the segmentation questions hinge on.

Reading: ~50 min Interactive: 2 widgets Source: Long et al. (2015) Fully Convolutional Networks · Ronneberger et al. (2015) U-Net · Noh et al. (2015) Deconvolution Network

01 · From classification to dense prediction

A label for every pixel

Semantic segmentation assigns a class to every pixel:

Segmentation task

I \in \mathbb{R}^{R\times C\times 3} \;\longrightarrow\; S \in \Lambda^{R\times C}

Every pixel of the $R\times C$ image gets a label from the class set $\Lambda$ — so training needs dense annotations.

Unsupervised vs semantic. Unsupervised segmentation just groups pixels by appearance (colour/texture) into coherent regions — superpixels, clustering — with no idea what the regions are. Semantic segmentation attaches a class from a fixed set to every pixel, and so needs dense annotations (e.g. COCO, where every pixel of every training image is labelled — extremely costly).

Semantic vs instance. Semantic segmentation gives every car the same “car” label; it does not separate car #1 from car #2 — that is instance segmentation. So a semantic network labels regions; it does not localise objects.

key

The patch-wise baseline (and why we abandon it)

The naive solution trains a classifier on patches, labelling each patch by its central pixel, then slides it over the image. It works but is hugely wasteful — overlapping patches recompute the same convolutions millions of times. Fixing that waste motivates everything below.

The core tension: a deep, low-resolution feature map knows what is in the image (semantics, large receptive field), while a shallow, high-resolution map knows where (location). A good segmentation network must combine both.

02 · FC-CNN: convolutionalizing a classifier

Turning dense layers into 1×1 convolutions

A classifier ends in Flatten → Dense, which fixes the input size and collapses space. But a dense layer is just a linear map — and a linear map over a feature volume is a convolution. A dense layer of $L$ outputs over an $N$ -channel feature vector equals a Conv2D with $L$ filters of size 1×1×N:

FC as 1×1 convolution

o_i = \mathbf{w}_i^\top \mathbf{s} + b_i = (\mathbf{w}_i \circledast \mathbf{s})(0,0) + b_i \;\equiv\; \text{Conv2D}(1{\times}1,\ \text{filters}=L)

The parameter count is identical; only the view changes. Two payoffs: the network now outputs a spatial heatmap (one map per class) instead of a single vector, and it accepts arbitrary input sizes. A bigger image simply yields a bigger heatmap — dense prediction in one forward pass, no recomputation over overlaps.

key

The flatten case

If the classifier flattens before its dense layer, the equivalent conv must have a kernel equal to the pre-flatten spatial size, with ‘valid’ padding, so it produces a single response (e.g. a 12×12×256 activation → Conv2D(256, kernel=12, valid)). Any further dense layers each map to their own 1×1 conv.

2024 · Fully-convolutionalization, true/false

A CNN of conv + maxpool + dense layers can always be turned into low-resolution class heatmaps (dense → 1×1 / ‘valid’ conv) — including nets with a flatten or a GAP head. Two traps: the “directly connected to the output” and “L/N filters” restrictions are false (multiple dense layers all convert), and a U-Net’s decoder cannot be trained from image-level labels — its expanding path needs dense masks. Also true: semantic segmentation is not object localisation.

The heatmap an FC-CNN produces is coarse — every pooling/stride step shrank it. Three strategies claw resolution back:

🔍 Direct upsampling

Bilinearly resize the low-res heatmap to image size. Cheap, but blurry — no learning.

🧵 Shift-and-stitch / à trous

Equivalent to dilated (“à trous”) convolutions that widen the receptive field without downsampling — dense outputs at full resolution.

🔼 Learnable upsampling

Train the decoder to upsample — transpose convolution (next section). The route U-Net takes.

key

FC-CNN's limitation

Convolutionalizing alone gets you low-resolution heatmaps and has no trainable upsampling. It is the right tool when you only have image-level labels — but for sharp, full-resolution masks you need a learned decoder.

03 · Upsampling & transpose convolution

Learning to grow the map back

To recover full resolution the decoder must upsample. Two options:

🔳 Max unpooling

Reuse the argmax locations stored during max-pooling; place values back there, zeros elsewhere. No learnable parameters.

🔼 Transpose convolution

A learnable upsample: each input value is scattered through a filter and the overlaps are summed. Stride controls the expansion factor.

Transpose convolution is the workhorse. With a 3-tap filter and stride 1, input $[I_1, I_2]$ expands to $[f_1 I_1,\ f_2 I_1 + f_1 I_2,\ f_3 I_1 + f_2 I_2,\ f_3 I_2]$ ; stride 2 spreads the contributions further for a larger expansion. It is exactly equivalent to upsample-then-convolve, but with learned filters.

Hands-on 1

Transpose convolution — learnable upsampling

A transpose conv scatters each input value through the filter and sums the overlaps. Tune the two inputs, the 3-tap filter, and the stride, and read the expanded output. Cells fed by two inputs (highlighted) are where overlap-sum — and checkerboard artifacts — appear.

I11.0

I20.6

f11.0

f20.7

f30.3

Input length

Output length

Overlap cells

Expansion

s·(N−1)+K

Try thisSwitch from stride 1 to stride 2: the output grows and the contributions spread apart, so fewer cells overlap. Uneven overlap between neighbouring cells is exactly the cause of the checkerboard pattern that plagues transpose-conv decoders.

TakeawayTranspose convolution is just a conv that increases spatial size with learned filters — equivalent to upsample-then-convolve. "Deconvolution" is a misnomer: it does not invert a convolution.

"Deconvolution" is a misnomer

Transpose convolution is also called fractionally-strided or backward-strided convolution, and — misleadingly — “deconvolution”. It does not invert a convolution; it is just a conv that increases spatial size. (Uneven overlapping sums also cause checkerboard artifacts.)

Losses. Segmentation loss is the per-pixel categorical cross-entropy summed over all pixels — so each image is already a mini-batch of pixel losses. For a pixel-wise regression task (e.g. predict height or denoise), the per-pixel loss becomes MSE, still summed over pixels.

Per-pixel loss

\hat{\theta} = \arg\min_\theta \sum_{x_j\in I} \ell(x_j,\theta,y_j)

There are two ways to feed this loss:

🧩 Patch-wise training

Sample patches and classify their centre pixel. Naturally balances classes and randomises mini-batches — but recomputes overlapping convolutions.

🖼️ Full-image training

One forward pass on the whole image; the loss is the sum of all pixel losses. Efficient and end-to-end — each image is already a mini-batch of pixels.

Full-image training loses the stochasticity and class-balancing that patch sampling gave for free, so it is recovered with a random pixel mask $M(x)$ (drop a random subset of pixels from the loss each step) and per-class loss weighting for imbalance.

why

Research application: kidney-biopsy segmentation from scribbles

Dense per-pixel labels are punishingly expensive for medical images. The lab’s kidney-biopsy work trains from sparse scribble annotations — a clinician draws a few strokes per class, and the loss is computed only on the labelled pixels (exactly the masking above). The network still learns to segment the whole image, turning a near-impossible labelling job into a few minutes of scribbling.

Q11

2026 · What can a same-size 3-channel (MSE) head do?

A U-Net-like net with a 3-channel, same-size output trained with MSE is doing per-pixel regression (image → image): drawing circles on a chessboard, recolouring portraits, predicting 3 real quantities per pixel (rainfall/pollution maps), or denoising all fit. What does not fit: anything needing per-pixel class probabilities (segmentation, X-ray bone/tissue/background → that is softmax + CE, not MSE), a single scalar/confidence, a few scalars (height/width/depth), or whole-image classification.

04 · U-Net

Encoder–decoder with concatenating skips

U-Net (Ronneberger et al., 2015) resolves the what/where tension with two paths and the skips that bridge them:

Contracting path (encoder): [3×3 Conv + ReLU] ×2 then 2×2 MaxPool; channels double at each downsample. Captures context (what).
Expanding path (decoder): 2×2 transpose conv (halves channels, doubles resolution), concatenate the matching encoder map, then [3×3 Conv + ReLU] ×2. Recovers location (where).

Hands-on 2

Skip connections — concatenate vs. add

A U-Net skip concatenates the encoder map onto the decoder (channels add up — always valid). A ResNet skip adds element-wise and therefore needs equal channels. Switch modes and channel counts to see the output depth and when Add breaks.

Encoder skip64

Decoder map64

encoder skip

64 ch

decoder map

64 ch

output

128 ch

Mode

Concat

Output channels

128

Needs equal C?

Concatenate stacks the channels: 64 + 64 = 128. A following conv mixes them. Always valid — no shape constraint.

Try thisIn Add mode set the encoder skip to 96 and the decoder to 64: it breaks. Switch to Concatenate and the same pair is fine, producing 160 channels — that is precisely why U-Net can bridge mismatched maps that ResNet's Add cannot.

TakeawayConcatenate (U-Net) raises the channel count and is always valid; Add (ResNet) needs equal channels (hence the 1×1 conv). Confusing the two is the classic model.summary() mistake.

key

Concatenate (U-Net) vs Add (ResNet)

U-Net skips concatenate the encoder feature maps onto the decoder, raising the channel count (a following conv then mixes them). This differs from a ResNet skip, which adds and therefore needs equal channels. Mixing these up is the classic model.summary() error (Chapter 10’s 2026 Q10).

The same conv-BN-ReLU block is reused in both paths; padding=1 with kernel 3 preserves $H,W$ . Downsampling, upsampling and skip-concatenation are interleaved between blocks:

class UNetBlock(nn.Module):
    def __init__(self, in_channels, out_channels):
        super().__init__()
        # padding=1 with kernel_size=3 preserves spatial dimensions
        self.conv1 = nn.Conv2d(in_channels, out_channels, 3, padding=1)
        self.bn1   = nn.BatchNorm2d(out_channels)
        self.conv2 = nn.Conv2d(out_channels, out_channels, 3, padding=1)
        self.bn2   = nn.BatchNorm2d(out_channels)
        self.relu  = nn.ReLU(inplace=True)
    def forward(self, x):
        x = self.relu(self.bn1(self.conv1(x)))
        x = self.relu(self.bn2(self.conv2(x)))
        return x

Exam-relevant properties of U-Net:

No fully-connected layers; the final layer is a 1×1 conv with #filters = #classes, then per-pixel softmax.
Fully convolutional → works on arbitrary input sizes (and tiles large images, predicting only where full context exists, so the output is smaller by a border).
Trained end-to-end on ~30 images with heavy elastic-deformation augmentation (the same warp applied to image and mask).
Uses a weighted loss: a class-balance term plus a border term that up-weights the thin background separating touching cells.
In full-image training, random masking $M(x)$ restores the mini-batch stochasticity that patch sampling would otherwise provide.

U-Net weighted loss

w(x) = w_c(x) + w_0\,\exp\!\left(-\frac{(d_1(x)+d_2(x))^2}{2\sigma^2}\right)

$w_c$ balances class frequencies; the exponential term up-weights the thin borders between touching cells ( $d_1,d_2$ = distances to the two nearest cells).

2025 · Semantic segmentation / U-Net true/false

True: the last layer is a 1×1 conv with #filters = #classes; class-specific loss weights are allowed; random masking mimics mini-batch stochasticity; encoder–decoder (contractive + expanding) is the dominant design; U-Net runs on images larger than training. False: “medical images only”, “photometric augmentation breaks the mask” (geometry is unchanged), “more filters → more spatial resolution”, “can count persons” (no instances), and “a patch classifier moved to an FCN is a U-Net” (that is only a coarse FCN, no learned decoder/skips).

Q10

2024 / 2026 · Fill the U-Net model.summary()

Both a TF residual net (2024 Q9) and a PyTorch U-Net (2026 Q10) ask you to complete the summary. Carry the Chapter 07/10 rules — Conv params $=(k_h k_w C_{in}+1)C_{out}$ , 'same'/padding=1 keeps $H,W$ , pooling halves them, BatchNorm = 4C (half non-trainable) — and remember the skip rows: a concatenate sets $C_{out}=C_{enc}+C_{dec}$ (the decoder conv then takes that larger input), whereas an Add keeps the channel count but needs the two branches equal.

05 · Exam intel

What the exam actually tests

Segmentation questions are fact-dense true/false plus a model.summary(). The recurring hinges: dense → 1×1 conv, transpose conv = learnable upsampling, and concatenate ≠ add.

A dense layer is a 1×1 convolution

A Dense layer of $L$ outputs over an $N$ -channel vector equals a Conv2D with $L$ filters of size 1×1×N — identical parameters. This makes a classifier fully convolutional: spatial heatmaps and any input size. (After a flatten, the first replacement conv is a ‘valid’ conv sized to the pre-flatten map.)

Transpose convolution is learnable upsampling

Transpose conv scatters each input through a learned filter and sums overlaps; stride sets the expansion. It equals upsample-then-convolve and is not a “deconvolution” (it does not invert a conv). Max unpooling is the parameter-free alternative.

Concatenate (U-Net) vs Add (ResNet)

U-Net skips concatenate → channels add up ( $C_{enc}+C_{dec}$ ), always valid. ResNet skips add element-wise → need equal channels (hence a 1×1 conv). This is the single most-tested segmentation fact.

U-Net facts worth memorising

Fully convolutional (any input size); final layer = 1×1 conv, #filters = #classes, per-pixel softmax; trained on ~30 images with elastic augmentation; border-weighted loss; semantic segmentation labels pixels but cannot count instances.

06 · Common mistakes

Where students get this wrong

"A U-Net skip adds the encoder map (like ResNet)"

U-Net concatenates — the channel count grows ( $C_{enc}+C_{dec}$ ) and a following conv mixes them. Only ResNet adds, which is why ResNet (not U-Net) needs equal channels and a 1×1 conv to match them.

"Transpose convolution inverts (de-convolves) a convolution"

It does not recover the original input; “deconvolution” is a misnomer. It is a conv that increases spatial size with learned filters — equivalent to upsample-then-convolve.

"Semantic segmentation can count the objects in a scene"

It labels each pixel (car / not-car) but does not separate car #1 from car #2 — that is instance segmentation. A semantic net cannot count individuals.

"Converting a classifier to an FCN gives sharp, full-resolution masks"

Convolutionalizing alone yields low-resolution heatmaps with no trainable upsampling. Sharp masks need a learned decoder (transpose convs + skips), i.e. a U-Net — which in turn needs dense pixel labels.

"A 3-channel MSE image output is a segmentation head"

Same-size 3-channel + MSE is per-pixel regression (image → image: denoise, recolour, quantity maps). Segmentation needs per-pixel class probabilities (softmax + cross-entropy), not MSE.

07 · Self-check

Can you answer these?

Four questions in the exact shapes the exam uses. Click an option for instant feedback.

To make a CNN classifier fully convolutional, a Dense layer with L outputs over an N-channel feature volume is replaced by…

Which statement about transpose convolution is correct?

A U-Net decoder receives a 64-channel upsampled map and concatenates the matching 64-channel encoder skip. What is the channel count entering the next conv, and does the skip require equal channels?

A network outputs a same-size, 3-channel image trained with MSE. Which task does it fit?

08 · Recap

One-screen summary

Chapter 11 — load-bearing ideas

Semantic segmentation labels every pixel but does not separate instances — it is not object localisation, and it cannot count.
FC-CNN: a Dense layer of $L$ outputs $=$ Conv2D with $L$ filters of size 1×1×N (same params); gives heatmaps and arbitrary input size, but only low-res with no trainable upsampling.
Transpose convolution is LEARNABLE upsampling ( $=$ upsample-then-conv); “deconvolution” is a misnomer. Per-pixel loss: summed cross-entropy (classification) or summed MSE (regression).
Recover resolution by direct upsampling, dilated (à trous) convs, or learnable upsampling. Train patch-wise (balanced, redundant) or full-image (efficient, + random masks / loss weighting); sparse scribbles compute loss only on labelled pixels.
U-Net skips CONCATENATE (raise channels $C_{enc}+C_{dec}$ ), unlike ResNet skips which ADD (need equal channels). Final layer = 1×1 conv with #classes filters.
U-Net is fully convolutional (any input size), trained on ~30 images with elastic augmentation and a border-aware weighted loss.