SLIM · Recommender Systems

01 · Stop choosing, start learning

From a heuristic to a learned similarity

Item-based CF picked a similarity formula. SLIM makes the matrix a thing you fit.

Recall the item-based recipe: build $S$ , then predict with $\hat R = R\,S$ . In Chapter 5 that $S$ came from a heuristic — cosine, but it could just as well have been Pearson, Jaccard, or Tversky. Nothing told us which makes the best recommendations. SLIM stops guessing the formula and learns $S$ from the data: the parameters are the entries of $S$ , fit by minimising a differentiable loss $E(S)$ .

Heuristic CF (Ch. 5)

S = \text{cosine}

. No training; you guess the formula. Fast, but fixed.

Learned CF — SLIM (Ch. 6)

S = \arg\min_S E(S)

. Trained on the data; the matrix adapts to what predicts ratings.

key

Same prediction, smarter S

$\hat R = R\,S$ is unchanged from Chapter 5 — SLIM only changes where $S$ comes from. Everything you know about item-based CF (model-based, $S$ precomputed, serves new users) still holds. SLIM upgrades the similarity from assumed to optimised.

!

Why the loss must be differentiable

We’d love to maximise Precision, Recall, MAP or AUC directly — but those depend on rank positions, which jump discretely, so they are not differentiable. With no usable gradient, SLIM optimises a smooth proxy: the MSE between $R$ and its reconstruction.

02 · Reconstruct the URM from itself

The objective: MSE reconstruction

Predict each rating column from all the others; minimise the squared error.

The prediction is literally item-based CF: $\hat r_{ui} = \sum_j r_{uj}\,s_{ji} = (R\,S)_{ui}$ . SLIM chooses $S$ to make this reconstruction as close as possible to the real ratings, by MSE over the observed ratings only: $E = \lVert R - R S\rVert^2$ . Read column by column, each item’s rating column is reconstructed from the other items’ columns — the URM explaining itself.

Hands-on

Learn one column of S · gradient descent

Fit the column from all-zero weights so R·S reconstructs the target item's rating column. Teal = the learned weight, grey = the cosine similarity from Chapter 5.

Learning rate γ0.010loss 0.04

iter 0MSE (peak 14.50)149

Column for The Avengers · ■ learned vs ■ cosine

TopGun0.260.96

MI0.691.00

Inter0.040.81

Martian0.000.70

Notting0.000.14

LoveAct0.080.16

LaLa0.000.20

TakeawaySLIM is the first model we *learn* by gradient descent. For Avengers it piles weight on MI and Top Gun and zeroes the rest — far sparser than cosine, which rated all four action films similar.

Formal Definition 6.1 — prediction & MSE loss

SLIM objective

\hat r_{ui} = (R\,S)_{ui} = \sum_{j} r_{uj}\,s_{ji}, \qquad E(S) = \sum_{(u,i)\in R^{+}} \big(r_{ui}-\hat r_{ui}\big)^2 = \lVert R - R S\rVert_F^2.

MSE is differentiable and interpretable; the trade-off is that it scores rating values pointwise rather than the ranking we actually serve (the job BPR takes over in Ch. 7).

03 · Two rules that keep S honest

The constraints: zero diagonal & non-negativity

Minimising the error naïvely has a trivial, useless solution. We forbid it.

Ask the optimiser to minimise $\lVert R - R S\rVert^2$ unrestricted and it spots a shortcut: set $S = I$ . Then $\hat R = R\,I = R$ , the error is exactly zero — and the model is useless, predicting each item by itself (0 for everything unseen). The cure is the zero diagonal constraint $\operatorname{diag}(S)=0$ : an item may not use its own rating to predict itself. SLIM adds non-negativity $S\ge 0$ : every learned weight is positive evidence.

Hands-on

What the constraints prevent

Minimising ‖R − R·S‖² with no restrictions has a trivial solution: S = I, each item predicting itself. The constraints forbid it.

train error 0.00 · learns real cross-item structure ✓ · but some weights go negative

TopGun0.42

MI0.63

Inter0.00

Martian-0.14

Notting-0.03

LoveAct0.09

Aveng (self)0.00

LaLa0.04

TakeawayZero training error is a red flag, not a trophy — here it means S = I. diag(S)=0 removes that escape hatch; S ≥ 0 keeps every weight as positive evidence.

!

Zero error is a red flag, not a trophy

If a model reaches zero reconstruction error on the training data, suspect a trivial solution (here $S=I$ ). The whole point of $\operatorname{diag}(S)=0$ is to remove that escape hatch so the optimiser must learn something that generalises.

Formal Definition 6.2 — constrained objective

Constrained

\min_{S}\; \lVert R - R S\rVert_F^2 \qquad \text{subject to}\quad \operatorname{diag}(S)=0,\quad S\ge 0.

Even so, $S$ has up to $\lvert\mathcal I\rvert^2$ parameters — far more than there are observed interactions — so we still need regularization.

04 · L1 for sparsity, L2 for stability

ElasticNet regularization

Too many parameters for the data — so penalise their size, two ways at once.

Even with the diagonal pinned to zero, $S$ has many more parameters than ratings to fit them. SLIM adds a regularization term penalising large weights, in two complementary flavours:

Ridge / L2 — penalty $\beta\lVert S\rVert_2^2$ . Shrinks weights smoothly toward zero; differentiable; keeps $S$ dense (weights get small but rarely exactly 0).
Lasso / L1 — penalty $\lambda\lVert S\rVert_1$ . Can force weights exactly to zero → a sparse model; not differentiable at 0 (solved numerically).

SLIM combines both — the elastic net: L1 gives a sparse, fast, storable $S$ ; L2 stabilises the solution when items are correlated.

Hands-on

ElasticNet · slide L1 and L2 on the learned S

The full learned S. Row = predictor item, column = target item; darker = larger weight. L1 drives weights to exactly zero (sparsity); L2 shrinks them smoothly (stability).

L1 λ3L2 β1non-zero: 32 / 56

i1i2i3i4i5i6i7i8TopGun·0.190.330.690.200.190.09MI0.39·0.290.070.740.02Inter0.060.04·0.04Martian0.390.18·0.290.03Notting0.12·0.460.90LoveAct0.32·0.06Aveng0.340.740.250.07·0.02LaLa0.100.270.670.38·

TakeawayAn L1-sparse S is small to store and fast to score, and its zeros are *learned* neighbour selection — like KNN, but the cut is found by the optimiser rather than imposed by hand.

key

Why sparsity is a feature, not just thrift

An L1-sparse $S$ is small to store and fast to score (most products are with zeros), and it is its own neighbour selection — like KNN in Chapter 4, but the zeros are learned rather than imposed by a top-K cut.

Formal Definition 6.3 — SLIM ElasticNet

ElasticNet

\min_{S}\; \lVert R - R S\rVert_F^2 \;+\; \beta\lVert S\rVert_2^2 \;+\; \lambda\lVert S\rVert_1 \quad\text{s.t.}\ \operatorname{diag}(S)=0,\ S\ge 0,

with $\beta,\lambda \ge 0$ tuned by validation. $\lambda=0$ recovers pure Ridge; $\beta=0$ recovers pure Lasso.

05 · Column by column, then back to CF

Training, parallelism & where SLIM sits

Each column is an independent regression — and the result is still item-based CF.

SLIM is trained by gradient descent: from an initial $S$ , repeatedly (1) sample an observed rating, (2) compute the gradient, (3) step each parameter against it by a small learning rate $\gamma$ — until convergence. The crucial structural fact: each column of $S$ is an independent problem (column $i$ only ever appears as “predict item $i$ from the others”), so SLIM decomposes into one regression per item, all solvable in parallel.

And where does SLIM land? Right back in Chapter 5’s family — still item-based, still model-based, still $\hat R = R\,S$ — only $S$ is now optimised instead of assumed. Feed the learned $S$ back through the prediction and Bob’s top unseen recommendation is The Martian, exactly what neighbourhood CF returned — but the weights were learned, not guessed.

key

The pattern that organises the rest of the course

SLIM is the first time we learn the model by minimising a loss with gradient descent — the same machine behind Matrix Factorization (Ch. 8) and the deep models later. Change the loss from MSE to a ranking loss and you get BPR (Ch. 7).

Formal Definition 6.4 — gradient & update

For an observed rating $(u,i)$ , the gradient w.r.t. weight $s_{ji}$ and the update with learning rate $\gamma$ :

SLIM update

\frac{\partial E}{\partial s_{ji}} = -2\,r_{uj}\,(r_{ui}-\hat r_{ui}) + 2\beta\, s_{ji} + \lambda\,\operatorname{sign}(s_{ji}), \qquad s_{ji} \leftarrow s_{ji} - \gamma\,\frac{\partial E}{\partial s_{ji}},\ \text{then clamp } s_{ji}\ge 0,\ s_{ii}=0.

Columns train independently and in parallel.

06 · Exam intel

What the exam tests

State the SLIM objective and prediction; explain why $\operatorname{diag}(S)=0$ is needed; contrast L1 vs L2; and carry out one gradient-descent update by hand.

Q

Worked question — one SLIM update by hand

SLIM is learning column $i$ = Avengers. Candidate weights put 0.5 on Top Gun and 0.5 on MI (others 0, $s_{\text{Aveng,Aveng}}=0$ ). Use Bob’s row [4,5,4,0,0,1,5,0].

(a) $\hat r = 4\cdot 0.5 + 5\cdot 0.5 = 4.5$ (only Top Gun and MI carry weight).
(b) error $e = 5 - 4.5 = 0.5$ , squared error $= 0.25$ .
(c) gradient $g = -2\,r_{u,\text{TopGun}}\,e = -2\cdot 4\cdot 0.5 = -4$ ; update $s \leftarrow 0.5 - 0.01\cdot(-4) = 0.54$ . The weight rises: the prediction was too low and Bob rates Top Gun highly.
Bonus. “1 on Avengers itself” is forbidden — it gives $\hat r = r$ , error 0, the trivial $S=I$ banned by $\operatorname{diag}(S)=0$ .

Traps: saying SLIM is a new prediction rule (it’s the same $\hat R=RS$ with a learned $S$ ); swapping L1 (sparsity) and L2 (shrink-only); forgetting $\operatorname{diag}(S)=0$ and “discovering” zero error.

07 · Exam · past papers

Past-paper questions

Past paper Practice Exam 4 · SLIM MSE, the zero diagonal, regularization

Q. How MSE is used as a loss in item-based CF and the overfitting problem; why the zero diagonal avoids trivial solutions; how regularization prevents overfitting and its effect on the parameters.

Model answer. MSE objective. SLIM reconstructs the URM through a learned item-item matrix $W$ : minimise $\tfrac12\lVert R - R\,W\rVert_F^2$ , the squared error between actual ratings and $\hat R = R\,W$ . Overfitting: $W$ has up to $\lvert\mathcal I\rvert^2$ free parameters, so it can memorise the training URM. Zero diagonal. Force $w_{ii}=0$ ; otherwise the trivial optimum $W=I$ reconstructs $R$ with zero error (pure leakage, generalises to nothing). Regularization. Add $\tfrac{\beta}{2}\lVert W\rVert_F^2 + \lambda\lVert W\rVert_1$ (ElasticNet): L2 shrinks weights smoothly, L1 drives many to exactly 0 — a sparse, efficient $W$ keeping only informative neighbours.

Past paper Practice Exam 4 · gradient descent for SLIM ElasticNet

Q. How gradient descent optimises parameters; the SLIM ElasticNet update; the iterative process.

Model answer. Gradient descent. Step parameters opposite the gradient: $\theta \leftarrow \theta - \eta\,\partial L/\partial\theta$ . Update (per column $w_i$ ). For $L=\tfrac12\lVert r_i - R\,w_i\rVert^2 + \tfrac{\beta}{2}\lVert w_i\rVert^2 + \lambda\lVert w_i\rVert_1$ : $\partial L/\partial w_i = -R^\top(r_i - R\,w_i) + \beta\,w_i + \lambda\,\operatorname{sign}(w_i)$ , then $w_i \leftarrow \max(0,\, w_i - \eta\,\partial L/\partial w_i)$ with $w_{ii}=0$ . Process. (1) sample interactions; (2) compute $\hat R = R\,W$ and the error; (3) compute the regularised gradient; (4) update and re-project onto the constraints. Repeat until validation stops improving; columns solve independently and in parallel.

08 · Self-check

Three questions before you move on

The fundamental difference between SLIM and neighbourhood item-based CF is:

Why does SLIM impose diag(S) = 0?

In SLIM's ElasticNet, the L1 (Lasso) term is responsible for:

09 · Recap

One-screen summary

Chapter 06 — load-bearing ideas

SLIM learns S, it doesn’t compute it: minimise $\lVert R - R S\rVert^2$ so $R\,S$ reconstructs the URM from itself. The prediction $\hat R = R\,S$ is still plain item-based CF.
Two constraints keep it honest: $\operatorname{diag}(S)=0$ kills the trivial $S=I$ ; $S\ge 0$ keeps weights as positive evidence.
ElasticNet shapes S: L1 → sparsity, L2 → stability; trained per column by gradient descent, fully parallel.
Update: $s \leftarrow s - \gamma(-2 r_{uj} e + 2\beta s + \lambda\operatorname{sign}(s))$ , then clamp.