Chapter 03

Evaluating Recommenders

Every chapter after this proposes an algorithm; this is the one that decides whether any of them work. What "good" means, how to split data without cheating, and the three metric families — error, classification and ranking — that turn a recommendation list into a single number.

Reading: ~60 min Interactive: 3 widgets Source: Polimi Recommender Systems 2024/25 — Evaluation & Relevance Metrics · Aggarwal, Recommender Systems — The Textbook, §7

01 · Many axes of quality

What does “good” even mean?

Relevance is only the headline. Real systems juggle a dozen quality axes.

Overall quality depends on three things together — the dataset, the algorithm, and the user interface. Accuracy is the headline, but a list that is accurate and monotonous, stale, or obvious is a bad product. The course names these indicators explicitly:

Relevance

Recommend items the user actually likes.

Diversity

Vary the list across genres; don’t repeat one niche.

Serendipity

Pleasantly surprise — relevant and unexpected.

Novelty

Surface items the user could not have found alone.

Coverage

Recommend most of the catalogue, not just the head.

Consistency & confidence

Don’t reshuffle wildly each visit; know how sure you are.

key

Relevance vs diversity vs serendipity

These pull against each other. The most “relevant” list might be ten near-identical action movies; injecting diversity and serendipity trades a little measured accuracy for a better experience. Beyond-accuracy metrics (catalogue coverage, intra-list diversity, novelty = average log-popularity) exist precisely to measure that trade.

There are two worlds in which to measure quality. Online evaluation puts the system in front of real users — direct feedback, A/B testing, controlled experiments. It measures what actually matters but is slow, costly, and risky. Offline evaluation replays a frozen dataset: no users, no UI, just the algorithm and the data. It is cheap and repeatable, which is why research lives here — and why the rest of this chapter does too.

02 · Splitting data without cheating

The offline pipeline & partitioning

Task → dataset → partitioning → metrics. Get the split wrong and every number lies.

An offline evaluation has four parts: the task, a representative dataset, partitioning into train/test, and metrics. The task is one of two:

Rating prediction — estimate the exact value (true 4, predicted 3.7). Judged by error metrics.
Top-N recommendation — produce a ranked list. Judged by classification & ranking metrics.

Exam favourite Detour: algorithm vs model

An algorithm is the process — the steps to train a model (dynamic, e.g. gradient descent). A model is the product — the learned parameters (static). Formally $\text{model} = f(\text{data})$ and $\hat r_{ui} = g(\text{model}, u)$ . For Global Effects (Ch. 2): the algorithm is “how to compute $\mu, b_i, b_u$ ”; the model is the actual values of $\mu, b_i, b_u$ .

Partitioning measures generalization — memorizing training data is worthless — so we hide a set of relevant items and check whether the model recovers them. Three strategies: hold-out of ratings (hide a random slice of cells; train and test users overlap), hold-out of users (set aside whole users — the new-user scenario), and a time-based split (train before a timestamp, test after).

No time travel

information leakage is using future data to predict the past. A random rating hold-out can leak: you might train on a rating a user gave after the one you are predicting. A time-based split avoids it — at the cost of a hard threshold (too short ⇒ constant retraining; too long ⇒ a stale model). Leakage is not the same as overfitting (interpolating the train set) — two distinct failures.

Hands-on

Hold-out splitter · train (X) vs hidden test (Z)

Filled cells are interactions. The splitter hides some as the test set Z(amber) and trains on the rest (X, teal). Switch the strategy to see how holding out ratings differs from holding out whole users.

Test fraction25%

i1i2i3i4i5i6i7i8Alice1111··1·Bob111··11·Carla11·11·11Dan··1·11·1Eva1··111·1Finn·11·1111

train X: 23test Z: 8

TakeawayHolding out users tests the new-user scenario (no overlap); holding out ratingsreuses the same users in both sets — easier, but it can leak future information into the past.

03 · Judging rating prediction

Error metrics: MAE, MSE, RMSE

For the rating-prediction task — and a warning about trusting them too far.

If the task is to predict a rating, the natural error of one prediction is $\lvert \hat r_{ui} - r_{ui}\rvert$ — e.g. $\lvert 4 - 3.7\rvert = 0.3$ . Averaged over the test set $T$ this gives MAE; squaring first gives MSE, and its square root is RMSE. RMSE punishes a few large misses far more than many small ones, so it is always $\ge$ MAE.

Hands-on

Error metrics · perfect → baseline predictor

Slide from a perfect predictor (r̂ = r) toward a baseline that predicts the global average (3.58) for everyone, and watch the three metrics climb — RMSE fastest, because the big misses dominate.

Predictor50%

MAE

0.83

MSE

0.83

RMSE

0.91

(user, item)	true r	pred	\|err\|	err²
Alice, MI	4	3.79	0.21	0.04
Bob, Aveng	5	4.29	0.71	0.50
Carla, Notting	1	2.29	1.29	1.66
Dan, LaLa	5	4.29	0.71	0.50
Eva, TopGun	1	2.29	1.29	1.66
Finn, Inter	2	2.79	0.79	0.62

TakeawayRMSE ≥ MAE always, and the gap widens as the predictor worsens — the squared term penalizes the rows where the truth is far from the average. But low RMSE never proves a good ranking.

Missing-As-Random is false

Error metrics are computed only on known test ratings, implicitly assuming the unknowns are missing at random (MAR). They are not — most unknown items are non-relevant and were never even shown. A model tuned to minimize RMSE is never judged on the strongly non-relevant items, so it is likely unsuitable for ranking. Low RMSE does not mean a good Top-N list.

Formal Definition 3.1 — error metrics

Over a test set $T$ of known ratings:

Error

\text{MAE} = \frac{1}{\lvert T\rvert}\sum_{(u,i)\in T} \lvert \hat r_{ui} - r_{ui}\rvert, \qquad \text{MSE} = \frac{1}{\lvert T\rvert}\sum_{(u,i)\in T} (\hat r_{ui} - r_{ui})^2, \qquad \text{RMSE} = \sqrt{\text{MSE}} \ge \text{MAE}.

04 · Did the right items make the cut?

Classification metrics: Precision & Recall

Treat the top-N list as a yes/no classifier and count hits.

For Top-N we recommend $N$ items and ask: how many were relevant? Sorting the catalogue into recommended/not × relevant/not gives the confusion matrix — TP (relevant & recommended, a hit), FP (recommended, not relevant), FN (relevant, missed), TN (correctly left out).

Precision@N asks “of what we recommended, how much was relevant?” — it divides hits by $N$ . Recall@N asks “of what was relevant, how much did we find?” — it divides hits by the total number of relevant items. Same numerator, different denominator.

Hands-on

Confusion matrix · precision / recall / F₁

Move the threshold. A higher threshold predicts “positive” less often — precision rises, recall falls. F₁ is their harmonic mean; the sweep shows where it peaks.

Threshold0.50

	pred 1	pred 0
actual 1	TP · 92	FN · 8
actual 0	FP · 16	TN · 84

Precision

0.852

Recall

0.920

F₁

0.885

F₁ sweep:0.00.670.10.690.20.710.30.760.40.870.50.880.60.860.70.620.80.330.90.111.00.06

TakeawayPrecision and recall trade off through the threshold; F₁ summarises the balance. Which to favour is a product decision — spam filters guard precision, cancer screens guard recall.

Missing-As-Negative

To compute these we must label the unknown items — and we simply call them all non-relevant (MAN). That is also false, but it works better than MAR for ranking, because it at least confronts the model with non-relevant items.

Formal Definition 3.2 — Precision & Recall

P/R

\text{P@}N = \frac{\#\text{relevant recommended}}{N} = \frac{TP}{TP+FP}, \qquad \text{R@}N = \frac{\#\text{relevant recommended}}{\#\text{relevant in test}} = \frac{TP}{TP+FN}.

05 · Order is everything

Ranking metrics: AP, MAP & ARHR

Precision@N ignores order inside the list. These metrics don’t.

Two lists can have identical Precision@N yet feel completely different: a user reads top-down, so a relevant item at rank 1 is worth more than the same item at rank 10. Average Precision sweeps $N$ down the list and averages the precision recorded at each relevant hit; for a user with a single relevant item it collapses to $\text{AP} = 1/\text{rank}$ . MAP averages AP across users; ARHR is a reciprocal-rank–weighted recall.

Hands-on

Top-N evaluator · one user, four recommenders

Relevant items are the ones the user rated ≥ 4. A recommender ranks all 8 movies; the cutoff N draws the line. Perfect stacks relevant items first (AP = 1); Anti-perfect buries them.

Cutoff N5

P@N 0.60R@N 0.60AP 0.56ARHR 0.24

1LaLa·

2TopGun✓ rel

3Notting·

4Martian✓ rel

5MI✓ rel

6LoveAct·

7Aveng✓ rel

8Inter✓ rel

Precision–Recall (N = 1…8)

TP 3FP 2FN 2TN 1

TakeawayPrecision@N ignores order inside the list; AP and ARHR reward relevant items placed early. Perfect → AP = 1; Shuffled → roughly constant precision ≈ #relevant / 8.

Formal Definition 3.3 — AP, MAP, ARHR

With $\text{rel}(k)\in\{0,1\}$ the relevance at rank $k$ and $R(k)$ the recall after $k$ items:

AP / MAP / ARHR

\text{AP}(N) = \sum_{k=1}^{N} P(k)\bigl(R(k)-R(k-1)\bigr) = \frac{1}{\#\text{rel}}\sum_{k=1}^{N} P(k)\,\text{rel}(k), \qquad \text{MAP} = \frac{1}{\lvert U\rvert}\sum_{u}\text{AP}_u, \qquad \text{ARHR} = \frac{1}{\#\text{rel}}\sum_{p=1}^{N} \frac{\text{rel}(p)}{p}.

06 · Exam intel

What the exam tests

Online vs offline; the precision/recall denominators; why error metrics mislead for ranking; and a hand computation of P@N, R@N and AP from a labelled list.

Worked question — P@5, R@5 and AP from a relevance pattern

A user has 3 relevant items in the test set. The top-5 list has relevance pattern [1, 0, 1, 0, 0].

P@5 $= 2/5 = 0.40$ (2 hits among 5 recommended).
R@5 $= 2/3 \approx 0.67$ (2 of the 3 relevant items found).
AP $= \tfrac{1}{3}\bigl(P(1)+P(3)\bigr) = \tfrac{1}{3}\bigl(\tfrac{1}{1}+\tfrac{2}{3}\bigr) = \tfrac{5}{9} \approx 0.56$ — precision is taken at each hit position (ranks 1 and 3), summed, and divided by the 3 relevant items.

Traps: dividing Recall by $N$ instead of by $\#\text{relevant}$ ; reporting RMSE as evidence of a good ranking; confusing leakage (future→past) with overfitting (memorizing the train set).

07 · Exam · past papers

Past-paper questions

Past paper RS Exam · 12 pts — precision, recall, the P–R curve, and AP

Q. Define precision and recall; plot each vs the cutoff K; plot precision vs recall and define AP; then: one user with one relevant item — algorithm A puts it at position 1, B at position 2 (two items each) — compute P and AP for both.

Model answer. $P@K=\frac{\#\text{rel in top-}K}{K}$ , $R@K=\frac{\#\text{rel in top-}K}{\#\text{rel}}$ . Recall is monotonically non-decreasing in K and tends to 1 as $K\to\lvert\mathcal I\rvert$ ; precision generally decreases with K (early ranks hold the best items, then you only add non-relevant ones). The P–R curve is a downward sawtooth. $\text{AP}=\frac{1}{\#\text{rel}}\sum_k P@k\cdot\text{rel}(k)$ . Exercise: both lists have one relevant item among two, so $P@2=\tfrac12$ for both. But AP uses the hit rank: A hits at rank 1 → $\text{AP}_A = 1$ ; B hits at rank 2 → $\text{AP}_B = 0.5$ . Same precision, but AP rewards A for ranking the relevant item higher.

Past paper FT-Feb26 · ~4 pts — error vs classification vs ranking metrics

Q. Describe the difference between error, classification, and ranking metrics. Give an example of each.

Model answer. Error metrics measure rating-prediction accuracy (predicted vs actual value); order-agnostic. Example: RMSE (or MAE). Classification metrics treat the top-N as a set of relevant/non-relevant items and ignore order within it. Example: Precision@K / Recall@K / F1. Ranking metrics reward placing relevant items higher. Example: MAP (or NDCG, MRR). Modern top-N evaluation favours ranking metrics because users only see the top of the list.

Past paper Practice Exam 1 · online/offline, P&R, and Missing-As-Negative

Q. Online vs offline evaluation (with an example of each); precision and recall; and the “Missing-As-Negative” (MAN) assumption and why it is used.

Model answer. Offline replays the recommender on a held-out split of logged interactions and computes metrics (cheap, reproducible, but a proxy) — e.g. leave-one-out Precision@10. Online runs the system on live users and measures real outcomes — e.g. an A/B test on click-through rate. $P@K=\frac{\#\text{rel in top-}K}{K}$ , $R@K=\frac{\#\text{rel in top-}K}{\#\text{rel}}$ . MAN: in offline top-N we have no true negatives, so every item the user did not interact with is assumed non-relevant. It lets classification/ranking metrics be computed on implicit data; the cost is a pessimistic bias — some “negatives” are simply items the user never saw, so measured precision underestimates true quality.

Past paper Practice Exam 3 · MAP, ARHR and A/B testing

Q. How is MAP calculated? What is the purpose of ARHR? What is an A/B test, and when is it used?

Model answer. MAP: average precision per user (the mean of precision values at that user’s relevant ranks), then average AP over users — $\text{MAP}=\frac{1}{\lvert U\rvert}\sum_u \text{AP}_u$ ; it rewards both finding relevant items and ranking them early. ARHR: the Average Reciprocal Hit-rate averages $1/\text{rank}$ of the hits, heavily rewarding a relevant item near the very top (rank 1 → 1, rank 2 → 0.5). A/B test: a controlled online experiment that splits live traffic between two variants and compares them on a business metric, giving a causal answer to “which is better” before full rollout.

08 · Self-check

Three questions before you move on

Recall@N divides the number of relevant recommended items by:

Why is a low RMSE a poor guarantee of a good Top-N ranking?

A user has exactly one relevant item, and the recommender places it at rank 3. Its Average Precision is:

09 · Recap

One-screen summary

Chapter 03 — load-bearing ideas

Three metric families: error (MAE/RMSE) for rating prediction; classification (Precision/Recall@N) and ranking (AP/MAP/ARHR) for Top-N.
Every metric assumes something false about the unknowns — MAR for error metrics, MAN for the rest. Error metrics are the worst proxy for ranking.
Order matters: AP rewards relevant items placed early; a lone relevant item at rank $k$ scores $1/k$ .
Recall’s denominator is $\#\text{relevant}$ , never $N$ ; and leakage (future→past) is not overfitting.