Chapter 12

Localization and Object Detection

By the end you can read an output head and name the task (regression vs detection), compute IoU, and trace the R-CNN family's evolution — why each version moved more of the pipeline into the network.

Reading: ~55 min Interactive: 2 widgets Source: Girshick et al. (2014) R-CNN · Girshick (2015) Fast R-CNN · Ren et al. (2015) Faster R-CNN · Redmon et al. (2016) YOLO · He et al. (2017) Mask R-CNN

01 · Localization & multi-task learning

One object: regress a box

Localization predicts a bounding box $(x, y, w, h)$ for a single object — a regression with 4 linear output neurons and an $\ell_2/\ell_1$ loss. Add a class label and you have a multi-task problem with two heads sharing a backbone:

Multi-task loss

\mathcal{L}(x) = \alpha\,\mathcal{S}(x) + (1-\alpha)\,\mathcal{R}(x)

$\mathcal{S}$ = softmax/cross-entropy (class), $\mathcal{R}$ = regression (box); $\alpha$ trades them off.

key

α is not an ordinary hyperparameter

Because $\alpha$ directly changes the loss definition, you cannot compare losses across $\alpha$ values or pick $\alpha$ by cross-validation — take it from the literature and judge it by a separate metric. Always better to train the two heads jointly.

Human pose estimation is the same regression idea scaled up: predict 2k coordinates for k body joints. Two landmark approaches show the design space:

🎯 DeepPose (2014)

Direct regression of all joint $(x,y)$ coordinates from the image, refined in a cascade of stages. Top-down, one person.

🤸 OpenPose

Bottom-up: predict joint heatmaps plus part-affinity fields that link joints, then assemble skeletons — handles multiple people in real time.

A recurring exam skill: read the output head and infer the task. A Dense(6, linear) head solves any task expressible as 6 real numbers (e.g. 2 scalars + a 4-coord box), but cannot do segmentation, detection, or variable-length outputs.

Q10

2024 / 2025 · What can a linear Dense(6) head do?

A linear Dense(6) + MSE head fits exactly tasks expressible as 6 homogeneous real numbers: weight + speed + a 4-coord box, 3 keypoints × 2 coords (ear/nose tips, pupil/tail centres), six counts, six ages, or vehicle dimensions + 3-D barycentre. It does not fit: per-pixel/image outputs (segmentation, weather maps), detection (variable #boxes), classification (needs softmax + CE), text/OCR, or a mismatched count (3 RGB values, 4 outputs). The output count must be exactly 6 regressions.

02 · Detection & IoU

Many objects, unknown count

Object detection outputs a variable number of (box, class) pairs per image — you don’t know in advance how many objects there are. The naive sliding-window solution (classify every crop at every scale) is hopeless: it recomputes overlapping convolutions and can’t pick a crop size.

To score a predicted box against the ground truth we use Intersection over Union:

IoU

\text{IoU} = \frac{\text{area}(\text{pred}\cap\text{GT})}{\text{area}(\text{pred}\cup\text{GT})}

Hands-on 1

Intersection over Union & NMS

IoU = area(∩) / area(∪) scores how well a predicted box (accent) matches the ground truth (green). Move and resize the prediction; at the NMS threshold 0.5 the box counts as a duplicate of the same object and is suppressed.

Intersection

6,912

Union

27,888

IoU

0.25

IoU 0.25 < 0.5: NMS would KEEP this box — it is treated as a distinct object.

Try thisLine the prediction up exactly on the ground truth: IoU → 1. Slide it half-off and IoU falls below 0.5, flipping the NMS verdict from "suppress" to "keep". That single threshold is what removes duplicate boxes around one object.

TakeawayIoU is trivial for axis-aligned boxes (just min/max corners) — which is exactly why rotated or elliptical anchors are awkward. NMS keeps the highest-objectness box and drops the rest above the IoU threshold.

IoU drives Non-Maximum Suppression (NMS): among overlapping boxes, keep the highest-objectness one and suppress the rest whose IoU with it exceeds a threshold — removing duplicate detections of the same object. (IoU is trivial for axis-aligned boxes, which is why rotated/elliptical anchors are awkward.)

03 · The R-CNN family

Moving the pipeline into the network

The R-CNN story is one of efficiency: each version absorbs another hand-built stage into the trainable network.

1️⃣ R-CNN (2014)

External Selective Search → ~2000 proposals, each warped and run through the CNN independently, then an SVM + box regressor. Not end-to-end. 84h train, 47s/image.

2️⃣ Fast R-CNN (2015)

One CNN pass on the whole image; proposals projected onto the feature map; ROI pooling → fixed-size vectors → FC heads with a multi-task loss (no SVM). End-to-end; bottleneck = proposal generation.

3️⃣ Faster R-CNN (2015)

Shared backbone + a learnable RPN → ~300 proposals → ROI pooling + detection head. ~0.2s/image.

The Region Proposal Network

The RPN slides a 3×3 conv over the shared feature map. At each location it scores k = 9 anchors (3 scales × 3 ratios), predicting:

RPN output per location

\underbrace{2k}_{\text{objectness (sigmoid)}} + \underbrace{4k}_{\text{box deltas }(\Delta x,\Delta y,\Delta w,\Delta h)}

Hands-on 2

RPN output shape — 2k objectness + 4k box deltas

A 3×3 conv slides over the shared feature map. At each location it scores k = scales × ratios anchors, predicting 2k objectness logits and 4k box deltas. Tune the anchor grid and the map size and watch the counts.

Scales3

Aspect ratios3

Map width40

Map height40

Classes (L−1)20

k anchors

2k objectness

4k box deltas

RPN ch / loc

total anchors = W·H·k = 40·40·9 = 14,400 → NMS + threshold → ~300 proposals

Detection head per proposal: 21 class scores (L = 20+1 with background) + 80 box coords = 4·(L−1) ⇒ 101 outputs. The background class gets no box.

Try thisSet scales = 3 and ratios = 3: the canonical k = 9 anchors, so the RPN emits 18 objectness + 36 delta channels per location. Note the proposal count stays ~300 no matter how many anchors you start with — that is NMS doing its job.

TakeawayAnchors are class-agnostic (they depend on scales and ratios, not the class count), so fine-tuning a detector for new classes never means changing anchors — only the detection head's L grows.

NMS + an objectness threshold reduce these to ~300 proposals. The detection head then returns L class scores + 4·(L−1) box corrections (no box for the background class).

2026 · R-CNN family, true/false

True: R-CNN uses a non-trainable Selective Search then crop/warp/classify; the RPN emits class-agnostic objectness; anchors give proposals the head refines; detection is multi-task (MSE + CE); YOLO is single-shot. Three traps: anchors are class-agnostic (fine-tuning for new classes never changes them), Fast R-CNN’s speed comes from ROI pooling on shared features (not separable convs), and Faster R-CNN is trained in multiple alternating stages, not a plain end-to-end pass. Also false: Mask R-CNN runs a U-Net per box (it adds a small mask head).

04 · YOLO, Mask R-CNN & FPN

Single-shot detection and instance segmentation

YOLO / SSD are region-free, single-shot detectors: reframe detection as one regression problem solved in a single forward pass. YOLO divides the image into a 7×7 grid; each cell predicts B anchors with $(dx, dy, dh, dw, \text{objectness})$ plus C class scores — an output tensor of 7×7×B×(5+C). SSD refines the idea by predicting boxes from multiple feature maps at different scales, so it handles a range of object sizes better than YOLO’s single grid. Both are faster than two-stage detectors, but less accurate on small/overlapping objects.

Mask R-CNN (2017) = Faster R-CNN + a parallel mask head that predicts a 14×14 mask for each of C classes, using ROI Align (bilinear) for better spatial alignment. It does instance segmentation (detection + per-instance mask).

key

Mask R-CNN is a mask head, not a U-Net per box

Mask R-CNN adds a small per-ROI mask branch on the shared features — it does not run an independent U-Net on each detected box.

Feature Pyramid Network (FPN) builds a multi-scale feature pyramid (bottom-up backbone + top-down upsampling + lateral 1×1 connections, combined by element-wise add) so detections at every level have strong semantics — crucial for small objects. FPN is a feature extractor, not a detector on its own.

key

Two-stage (R-CNN family) = more accurate; single-shot (YOLO/SSD) = faster. FPN + Faster R-CNN is the standard accuracy/speed sweet spot.

Where this stack goes in practice — the detection/segmentation toolkit underpins much of the lab’s applied research:

🚗 Autonomous driving

3D object detection fusing LiDAR point clouds with camera images — boxes in 3D, not just on the image plane.

🆔 Object re-identification

Match the same instance (a person, a vehicle) across cameras — a latent-embedding similarity task on detected crops.

🎞️ Video object segmentation

Propagate a mask through time, segmenting an object across every frame of a clip.

🧬 Generative augmentation

Synthesise training images (e.g. for scarce biomedical datasets) to enlarge and balance the training set — data scarcity meets generation.

05 · Exam intel

What the exam actually tests

Detection is tested through output-head reading, the RPN shape, IoU/NMS, and a dense R-CNN true/false. Know the anchor arithmetic and the class-agnostic anchor fact cold.

Read the head, name the task

A fixed linear Dense(k) + MSE head does k-value regression only — box coords, keypoints, counts, scalars. It can never be segmentation (per-pixel softmax), detection (variable #boxes), or classification (softmax + CE). Match the loss/activation to the output structure.

IoU and NMS

$\text{IoU} = \text{area}(\cap)/\text{area}(\cup)$ scores box quality. NMS keeps the highest-objectness box and suppresses overlapping boxes whose IoU exceeds the threshold — that is how duplicate detections of one object are removed.

RPN output = 2k + 4k

Over $k =$ scales × ratios anchors, the RPN emits 2k objectness (object/not, sigmoid) + 4k box deltas $(\Delta x,\Delta y,\Delta w,\Delta h)$ per location; NMS cuts $W{\cdot}H{\cdot}k$ anchors to ~300 proposals. The detection head returns L scores + 4(L−1) box coords (no box for background).

The three R-CNN traps

Anchors are class-agnostic (depend on scale/ratio, not #classes — fine-tuning never changes them); Fast R-CNN’s speed is ROI pooling on shared features, not separable convs; Faster R-CNN trains in multiple alternating stages, not plain end-to-end. Mask R-CNN adds a mask head, not a per-box U-Net.

06 · Common mistakes

Where students get this wrong

"Pick the multi-task weight α by cross-validation"

$\alpha$ redefines the loss, so losses at different $\alpha$ are not comparable and CV cannot select it. Take $\alpha$ from the literature and judge the model by a separate metric; train both heads jointly.

"A linear Dense(6) head can do detection or segmentation"

It does 6 fixed regressions only. Detection needs a variable number of boxes; segmentation needs per-pixel class probabilities. Neither fits a fixed 6-scalar MSE head.

"Fine-tuning a detector means changing the anchors for the new classes"

Anchors are class-agnostic — they depend on scales and aspect ratios, not the class count. Only the detection head’s number of class scores changes.

"Fast R-CNN is fast because of separable convolutions"

Its speed-up comes from running the backbone once on the whole image and using ROI pooling on the shared feature map — not from MobileNet-style separable convolutions.

"Mask R-CNN runs a U-Net on each detected box"

It adds a small per-ROI mask head on the shared ROI-Align features — one lightweight branch, not an independent U-Net per box.

07 · Self-check

Can you answer these?

Four questions in the exact shapes the exam uses. Click an option for instant feedback.

A network ends in a linear Dense(6). Which task can it be trained for?

Non-Maximum Suppression uses IoU to…

With k = 9 anchors, how many values does the RPN predict per feature-map location?

Which statement about the R-CNN family is TRUE?

08 · Recap

One-screen summary

Chapter 12 — load-bearing ideas

Localization = regress 4 box coords (linear). Multi-task loss $\mathcal{L} = \alpha\mathcal{S} + (1-\alpha)\mathcal{R}$ ; $\alpha$ reshapes the loss, so it is NOT tuned by cross-validation.
Read the output head to name the task: a fixed Dense(k, linear) head does k-value regression — never segmentation/detection/variable-length output.
IoU = intersection/union scores box quality and drives NMS (suppress overlapping duplicates by objectness + IoU).
R-CNN → Fast → Faster: each step absorbs a hand-built stage. The RPN outputs 2k objectness + 4k box deltas over k = 9 anchors → ~300 proposals; the head returns L scores + 4(L−1) boxes.
Anchors are class-agnostic; Fast R-CNN’s speed is ROI pooling; Faster R-CNN is multi-stage trained (not plain end-to-end). Mask R-CNN adds a mask head (not a per-box U-Net); YOLO/SSD are single-shot; FPN adds multi-scale features.