Chapter 11

Deep Learning for RecSys

KNN, SLIM and MF are linear at heart. Deep learning adds stacked non-linear layers that learn their own features. Three architectures dominate: the autoencoder (compress a profile, reconstruct it, read off the blanks), the two-tower model (one tower per side, joined at a dot product), and the variational autoencoder. Strip the non-linearities and a shallow autoencoder collapses to a factorised item–item similarity — MF and item-CF are special cases.

Reading: ~65 min Interactive: 1 widgets Source: Polimi Recommender Systems 2024/25 — Deep Learning for RecSys · Steck, Embarrassingly Shallow Autoencoders for Sparse Data (WWW 2019) · Liang, Krishnan, Hoffman & Jebara, Variational Autoencoders for Collaborative Filtering (WWW 2018) · Ferrari Dacrema, Cremonesi & Jannach, Are We Really Making Much Progress? (RecSys 2019)

key

The big idea

A neural network is a stack of fully-connected layers $x_o = f(x_i\,W + b)$ . Stack enough with non-linear activations and the model learns its own features. Three architectures map this to recommendation: the autoencoder encodes the user profile $r_u$ into a compressed embedding, decodes back, and fills in the zeros; the two-tower model learns separate user and item embeddings then dots them; the variational autoencoder encodes a distribution instead of a point. Remove the non-linearities and both collapse to familiar forms — MF and item–item CF.

01 · Layers, weights, activations

The fully-connected layer

One equation, three components: a linear transform, a bias, and a non-linearity.

A fully-connected layer (a “dense” layer) takes a weighted sum of its inputs, adds a bias, and applies a non-linear activation function $f$ :

Fully-connected layer

x_o = f\!\big(x_i\,W + b\big),\qquad x_i\in\mathbb{R}^n,\; x_o\in\mathbb{R}^m,\; W\in\mathbb{R}^{n\times m},\; b\in\mathbb{R}^m.

$W$ and $b$ are the learnable parameters; the activation $f$ is the essential source of non-linearity — without it, stacking layers collapses to a single linear transform. Two common choices:

Sigmoid (logistic)

\sigma(x)=\tfrac{1}{1+e^{-x}}

, squashing to

(0,1)

. Smooth and differentiable, but suffers vanishing gradients for large

\lvert x\rvert

ReLU

\mathrm{ReLU}(x)=\max(0,x)

. Cheap, no vanishing gradient for positive inputs, sparse activations — the default for hidden layers.

An MLP (multilayer perceptron) chains such layers, $x_h = f_h(x_i\,W_h + b_h)$ then $x_o = f_o(x_h\,W_o + b_o)$ . “Deep” means many layers; depth lets the network learn features from raw input rather than requiring hand-crafted ones. By contrast a neural model is any composition of parameterised functions — including linear regression and matrix factorization (one layer, linear activation). The term “deep learning” is often misused to mean “any neural model with several layers,” even when the model does not truly learn its own representations. Specialised layers exist for specific data — convolutional for images, recurrent (LSTM) for sequences, attention for sets — though the lecture notes that LSTM is not on the exam.

key

The ingredients are few

To build a neural recommender you need: (1) an architecture (which layers, how connected), (2) a loss function (MSE, BPR, BCE, contrastive…), and (3) a sampling strategy (which users, which positives/negatives). Everything else — gradients, updates — the framework computes. The art is in the architecture and the loss, not the optimiser.

02 · BCE loss

Binary Cross Entropy

The default loss when treating implicit feedback as binary classification.

When ratings are implicit ( $r_{ui}\in\lbrace 0,1\rbrace$ — interacted or not) and the model outputs a probability $\hat p_{ui}=\sigma(\hat r_{ui})$ , the standard loss is binary cross entropy (BCE):

BCE loss

\mathcal{L}_{\text{BCE}} = -\frac{1}{N}\sum_{(u,i)\in\mathcal{S}}\Big[\, r_{ui}\log\hat p_{ui} + (1-r_{ui})\log\big(1-\hat p_{ui}\big)\,\Big].

Sampling matters. The URM has too many zeros to train on all of them. Two dangers: training on only positives never teaches the model to rank (it just learns to score everything high); training on all ground truth lets the negatives (over 99% of the matrix) drown the positives. The fix is to subsample — draw each training example as positive with roughly 50% probability, negative otherwise.

key

BCE is not just for deep models

BCE can replace MSE in any algorithm we’ve seen — SLIM, MF, and the rest. The difference is that BCE punishes confident wrong predictions far more harshly than MSE (the log blows up when $\hat p$ is near 0 for a positive, or near 1 for a negative), which suits implicit, binary data.

03 · Compress, reconstruct, predict

Autoencoders for recommendation

Feed a user’s entire profile in, reconstruct it, and the reconstructed zeros are the predictions.

An autoencoder learns a compressed latent representation (the embedding) of its input that allows reconstructing the input as accurately as possible. Two networks — an encoder $g_e$ and a decoder $g_d$ :

Encoder / Decoder

e_u = g_e(r_u),\qquad \hat r_u = g_d(e_u).

For recommendation, the pipeline is:

Sample a user profile $r_u$ — the full URM row, with zeros for unrated items.
Encode $e_u = g_e(r_u)$ — compress the profile into a low-dimensional embedding of the user’s interests.
Decode $\hat r_u = g_d(e_u)$ — reconstruct the full-length profile; unrated items get non-zero scores.
Rank items by $\hat r_u$ — the filled-in zeros are the recommendations.

The collaborative signal is indirect: users with similar interaction patterns get similar embeddings, because the encoder and decoder are trained to reconstruct any user profile well. There is no explicit similarity computation — the network discovers communities on its own. The natural training objective is a reconstruction loss, MSE $\lVert r_u - \hat r_u\rVert^2$ or BCE.

key

Autoencoders are model-based

Unlike memory-based KNN, an autoencoder does not store the URM at prediction time. If a new user appears or an existing user adds interactions, just feed the updated profile through the trained encoder → decoder — no retraining required (though retraining may improve accuracy). This is the same advantage MF (Ch. 8) has over neighbourhood methods.

04 · Strip the non-linearities

Shallow autoencoders as item–item similarity

Remove activation functions and biases, and the autoencoder becomes a factorised similarity matrix.

Take a shallow autoencoder with no hidden layers, embedding size $K$ , linear activation $f=I$ , and zero biases:

Shallow AE (linear)

e_u = r_u\,W_e,\qquad \hat r_u = e_u\,W_d = r_u\,W_e\,W_d,\qquad W_e\in\mathbb{R}^{I\times K},\; W_d\in\mathbb{R}^{K\times I}.

The product $W_e\,W_d$ is an $I\times I$ matrix. Writing $S = W_e\,W_d$ , we get $\hat r_u = r_u\,S$ — exactly the item–item similarity form of Ch. 5/6. This factorised $S$ is an asymmetric similarity (separate encoder and decoder weights). Tie the weights so $W_d = W_e^\top$ and it becomes symmetric:

Shallow AE (symmetric)

S = W_e\,W_e^\top,\qquad \hat r_u = r_u\,S.

This is exactly how PureSVD (Ch. 8) works — the truncated SVD gives $R\approx R\,V_K V_K^\top$ . The lecture’s conclusion: MF is a shallow autoencoder, and an autoencoder with non-linearities is a non-linear generalisation of MF.

Asymmetric S

S = W_e W_d

(separate weights). Equivalent to Asymmetric SVD (Ch. 8): the user profile is the input, the item–item

S

is factorised.

Symmetric S

S = W_e W_e^\top

(tied weights). Equivalent to PureSVD:

S

is positive semi-definite by construction, rank at most

K

Hands-on

Shallow autoencoder · compress a profile, read off the blanks

The encoder compresses Bob's profile to K numbers; the tied decoder reconstructs every item. Blanks the model scores high are the recommendations. Raise K to sharpen the reconstruction — and watch it start to overfit.

Embedding K2MSE 0.22

TopGunMIInterMartianNottingLoveActAvengLaLaactual454··15·AE K=24.44.14.03.8-0.01.14.11.1

Recommendations for Bob (K=2, symmetric AE)

1Martian3.83

2LaLa1.10

3Notting-0.01

TakeawayWith linear activations and tied weights the whole network collapses to r̂_u = r_u·S with S = W_e W_eᵀ — a rank-K item–item similarity, exactly PureSVD. Non-linear activations would generalise it, but on a sparse URM the extra capacity mostly overfits — the whole point of this chapter's reproducibility warning.

key

Adding depth changes what S can express

The linear shallow AE is constrained to rank- $K$ similarity. Deep autoencoders with non-linear activations can represent any continuous mapping from profiles to scores — but at the cost of harder optimisation, more parameters, and a real risk of overfitting on sparse data.

05 · Embarrassingly shallow

EASE-R

A closed-form autoencoder — no gradient descent, yet competitive with deep models.

EASE-R (Embarrassingly Shallow Autoencoders for Sparse Data, Steck 2019) is an item-based similarity model whose objective looks like an autoencoder: reconstruct the input, $R\approx R\,S$ , with the twist that an item may not reconstruct itself.

EASE-R objective

S^{*} = \arg\min_S\; \lVert R - R\,S\rVert_F^2 + \lambda\lVert S\rVert_F^2,\qquad \text{s.t.}\;\operatorname{diag}(S)=0.

The constraint $\operatorname{diag}(S)=0$ prevents the trivial self-reconstruction; it is enforced with Lagrange multipliers, and — remarkably — the problem has a closed-form solution, no gradient descent needed.

Pros

Fast to train (closed form). Highly effective — competitive with deep models. One real hyperparameter,

\lambda

Cons

Computing

P=(R^\top R + \lambda I)^{-1}

is memory-intensive —

O(I^2)

storage. Infeasible for very large catalogues.

The paper calls it an autoencoder because the loss minimises $\lVert R - R\,S\rVert_F$ — reproducing the input as its output — despite there being no explicit encoder/decoder architecture. The “autoencoder” label is conceptual, not architectural.

Formal Definition 11.1 — EASE-R closed form

Solving the Lagrangian gives a four-step recipe:

EASE-R solution

P = (R^\top R + \lambda I)^{-1},\qquad S^{*} = I - P\,\operatorname{diagMat}\!\big(1 \oslash \operatorname{diag}(P)\big).

(1) Compute $P=(R^\top R + \lambda I)^{-1}$ . (2) Take the diagonal of $P$ and its element-wise inverse $1/\operatorname{diag}(P)$ . (3) Scale the columns of $P$ by that inverse. (4) Subtract from $I$ to zero the diagonal. No iteration — just one matrix inverse.

06 · Robustness and uncertainty

Denoising and Variational Autoencoders

Two upgrades: corrupt the input during training, or encode a distribution instead of a point.

A denoising autoencoder (DAE) addresses sparse, mostly-zero profiles: the encoder may learn poor embeddings and the decoder may not know unfamiliar regions of the space. The fix is to corrupt the input and train the network to reconstruct the clean profile. The classic corruption is salt & pepper noise: randomly drop a fraction of positive interactions, and randomly add a few false positives. Both encoder and decoder then see many more profiles, generalising better. Formally, minimise $\lVert r_u - \hat r_u^{\,\text{noise}}\rVert$ with $\hat r_u^{\,\text{noise}} = g_d(g_e(\tilde r_u))$ , where $\tilde r_u$ is the corrupted profile.

A variational autoencoder (VAE) encodes the input not as a single point but as a probability distribution: the encoder outputs a mean $\vec\mu$ and standard deviation $\vec\sigma$ , a sample $e\sim\mathcal{N}(\vec\mu,\vec\sigma)$ is drawn and decoded. Sampling is not differentiable, so the reparametrisation trick moves the randomness aside:

Reparametrisation trick

\vec e = \vec\mu + \vec\sigma \odot \vec\epsilon,\qquad \vec\epsilon \sim \mathcal{N}(0,1).

Now gradients flow through the deterministic $\vec\mu$ and $\vec\sigma$ , while the randomness sits in $\vec\epsilon$ , which is not learned. The VAE is a generative model: it assumes the data $x$ is sampled from $P_\theta(x)=\int P_\theta(x\mid z)\,P(z)\,dz$ with prior $P(z)\sim\mathcal{N}(0,I)$ . Mult-VAE (Liang et al., 2018) applies this to collaborative filtering with a multinomial likelihood — highly effective at limited cost (but it wants a GPU).

key

Why encode a distribution?

A point embedding forces one representation per profile; a distribution captures uncertainty — the model knows what it doesn’t know. Sampling during training acts like built-in data augmentation, making the decoder robust. The DAE achieves a similar effect more simply, at the input level (corruption) rather than the embedding level.

07 · One tower per side

The Two-Tower Model

Separate user and item encoders, joined at a dot product.

The two-tower model has two inputs — a user input and an item input — each processed by its own “tower” of layers up to a final embedding; the two embeddings are combined (typically by dot product):

Two-Tower prediction

\hat r_{ui} = e_u\cdot e_i,\qquad e_u = \text{UserTower}(x_u),\; e_i = \text{ItemTower}(x_i).

The inputs can be one-hot IDs, full profiles (a URM row / column), or unstructured data (text via an LLM, images, behavioural sequences via an RNN). Unlike autoencoders, a two-tower model predicts $\hat r_{ui}$ directly, so any loss applies — MSE, BPR, BCE, contrastive. With a ranking loss you must score several sampled positives and negatives per step; an autoencoder gives predictions for all items by construction.

Two-tower is MF in disguise. With no hidden layers, linear activation, zero bias, and one-hot inputs, each tower just selects a row of its weight matrix:

Two-Tower = MF

e_u = x_u\,W_U = W^{(U)}_u,\qquad e_i = x_i\,W_I = W^{(I)}_i,\qquad \hat r_{ui} = W^{(U)}_u\cdot W^{(I)}_i.

Those rows are exactly the user and item latent factors from MF (Ch. 8); adding non-linear layers makes this a non-linear MF. With one-hot input the model is memory-based — a new user has no row without retraining. Replace the one-hot with the full user profile and it becomes Asymmetric SVD, hence model-based: a new user simply passes through the encoder.

key

Two-tower is the workhorse of industry

The architecture is simple, modular (swap the user tower without touching the item tower), and accepts rich inputs (text, images, sequences). Pre-compute the item embeddings offline; at serving time you run one forward pass through the user tower and a dot product against the item table.

08 · The reproducibility crisis

All is great… but!

Deep learning for RecSys: powerful in principle, underwhelming in reproducibility.

The lecture closes on a sobering result from Ferrari Dacrema et al.’s systematic analysis:

Of 18 deep-learning algorithms presented at top conferences, only 7 could be reproduced with reasonable effort — and 6 of those were often outperformed by comparably simple heuristics, e.g. nearest-neighbour or graph-based methods.

The recurring concerns for deep learning in RecSys:

Reproducibility — many papers omit code, use undocumented tuning, or rely on non-deterministic GPU training.
Overfitting — deep models have far more parameters than the (over 99% sparse) observed interactions; the risk is much higher than in vision or NLP, where data is abundant.
Computational cost — training wants GPUs and inference can be slow; simple baselines (KNN, SLIM, EASE-R) run on a CPU in seconds.
Diminishing returns — the added depth often gives only marginal gains over a well-tuned shallow model — sometimes none.

The lesson is not “don't use deep learning”

It is always compare against simple baselines. EASE-R (a closed-form linear model) consistently beats many deep autoencoder variants; a well-tuned item-CF or SLIM may out-rank a complex two-tower model. Deep learning shines when the input is unstructured (text, images, sequences) or the data is genuinely large — but on the standard CF benchmark, the gains are often marginal.

09 · Exam intel

What the exam tests

Write the fully-connected layer $x_o=f(x_i\,W+b)$ ; give the sigmoid and ReLU formulas; describe the autoencoder CF pipeline; show that a shallow linear AE gives $\hat r_u = r_u\,W_e\,W_d$ (factorised item–item $S$ ); explain the two-tower model and its equivalence to MF with one-hot input; state the EASE-R objective and closed form; the DAE noise types; the VAE reparametrisation trick; and the reproducibility critique.

Worked question — shallow AE and model-based vs memory-based

A shallow autoencoder has $K=3$ , linear activations, zero biases, encoder $W_e\in\mathbb{R}^{I\times K}$ and decoder $W_d\in\mathbb{R}^{K\times I}$ . (a) Structure of the effective item–item similarity $S$ ? (b) If $W_d=W_e^\top$ , what does that impose on $S$ ? (c) Why is the autoencoder model-based while two-tower with one-hot input is memory-based?

(a) $S = W_e\,W_d\in\mathbb{R}^{I\times I}$ — a factorised asymmetric item–item similarity, scoring $\hat r_u = r_u\,S$ .
(b) $S = W_e\,W_e^\top$ : symmetric and positive semi-definite, rank at most $K=3$ (the PureSVD case).
(c) The autoencoder takes the full profile as input, so a new user passes through the trained encoder. Two-tower with one-hot input selects a row of $W_U$ — a new user has no row (memory-based). Feeding the full profile instead makes two-tower model-based (Asymmetric SVD).

Traps: confusing “neural” (any differentiable composition) with “deep” (learns its own features — MF is neural but not deep); forgetting $S=W_eW_d$ is asymmetric unless weights are tied; calling autoencoders memory-based; and thinking $\operatorname{diag}(S)=0$ in EASE-R is done by zeroing after training rather than via Lagrange multipliers.

10 · Exam · past papers

Past-paper questions

Past paper AT-Jan26 · Autoencoder recommender; SLIM as a special case (5 pts)

Q. Autoencoder with one hidden layer: draw it (1); write each layer’s equations (2); why and when item-based CF (e.g. SLIM) is a special case (2).

Model answer. Architecture. Input = the user’s rating vector $r_u\in\mathbb{R}^{\lvert I\rvert}$ → a smaller bottleneck → output of size $\lvert I\rvert$ reconstructing the ratings. Equations. Encoder $h=f(W\,r_u+b)$ ; decoder $\hat r_u=g(W'\,h+b')$ ; train on observed entries, $\min\sum_u\lVert (r_u-\hat r_u)\odot m_u\rVert^2$ with mask $m_u$ . SLIM as special case. A linear, single-layer AE with identity activation reconstructing the URM is $\hat R = R\,W$ with $\operatorname{diag}(W)=0$ — exactly item-based CF / SLIM. The autoencoder generalises it with non-linearity and a bottleneck.

Past paper AT-Feb26 · Two-tower; model-based vs memory-based (5 pts)

Q. Two-tower recommender, each tower one hidden layer: draw it (1); equations per tower (2); when it is model-based or memory-based (2).

Model answer. Architecture. Two independent towers — user and item — each a fully-connected net with one hidden layer; outputs combined by a dot product. Equations. $p_u=f(W_u\,x_u+b_u)$ , $q_i=f(W_i\,x_i+b_i)$ , $\hat r_{ui}=p_u\cdot q_i$ (optionally through a sigmoid). Model- vs memory-based. Fundamentally model-based — predictions come from learned weights. With one-hot ID inputs the towers are embedding tables (transductive: a new user needs retraining); with feature/history inputs a tower can embed an unseen user at inference (inductive).

Past paper FT-Jan26 · MF as a special case of two-tower (6 pts)

Q. Two-tower (one hidden layer per tower): draw it (1); equations (2); why and when MF is a special case (2); is it model- or memory-based (1)?

Model answer. Equations as above. MF as special case. Make both towers linear (identity activation) and feed one-hot IDs: each tower reduces to an embedding lookup, so $p_u,q_i$ are the latent factor rows and $\hat r_{ui}=p_u\cdot q_i$ is exactly Matrix Factorization. Two-tower generalises MF with non-linear towers and side-feature inputs. Model-based — recommendations come from learned parameters (transductive with ID inputs, inductive with features).

Past paper Practice Exam 2 · NN layer, activations, BCE (5 pts)

Q. The structure of a NN layer (2); how sigmoid and ReLU add non-linearity (1); how BCE is used in a recommender (2).

Model answer. Layer. $z=Wx+b$ , then a non-linear activation $a=\phi(z)$ ; stacking builds a deep net. Sigmoid & ReLU. $\sigma(z)=1/(1+e^{-z})\in(0,1)$ — smooth, probability-like, but saturates (vanishing gradients); $\mathrm{ReLU}(z)=\max(0,z)$ — cheap, non-saturating for positive inputs, sparse. Without them a stack collapses to one linear map. BCE. For implicit 0/1 targets interpret $\hat y_{ui}=\sigma(\text{score})$ as $P(\text{interaction})$ and minimise $-\sum[y\log\hat y + (1-y)\log(1-\hat y)]$ , with sampled negatives.

Past paper Practice Exam 2 · Autoencoders, embeddings, VAE (5 pts)

Q. How autoencoders are used for RS (2); the concept of an embedding (1); how VAEs differ and the role of the reparametrisation trick (2).

Model answer. AEs for RS. Feed a user’s partial rating vector, compress to a latent code, reconstruct the full vector; the reconstructed scores for unrated items are the recommendations (train on observed entries). Embedding. The bottleneck code — a low-dimensional dense vector summarising preferences. VAE vs AE. A standard AE maps an input to a single point; a VAE’s encoder outputs a distribution $\mathcal{N}(\mu,\sigma^2)$ , samples $z$ , and adds a KL term pulling the latent toward a prior — a smoother, generative space (e.g. Mult-VAE). The reparametrisation trick writes $z=\mu+\sigma\odot\varepsilon,\ \varepsilon\sim\mathcal{N}(0,1)$ so the stochastic node moves outside the network and gradients backprop through $\mu,\sigma$ .

Past paper Practice Exam 5 · DAE, two-tower, shallow AE → S (5 pts)

Q. Denoising autoencoders and their training (2); what a two-tower model is (1); how a shallow autoencoder yields an item–item similarity (2).

Model answer. DAE. Corrupt the input (drop interactions / add noise) and train to reconstruct the clean vector — prevents the trivial identity map, adds robustness, mirrors the missing-data nature of implicit feedback. Two-tower. Separate user and item nets producing embeddings whose dot product scores a pair, $\hat r_{ui}=p_u\cdot q_i$ — efficient because item embeddings precompute for nearest-neighbour retrieval. Shallow AE → S. A single linear layer reconstructing the URM, $\hat R = R\,W$ with a zero diagonal, learns a weight matrix $W$ that is an item–item similarity — the same object as SLIM / EASE-R.

11 · Self-check

Three questions before you move on

A shallow autoencoder with linear activations, zero biases, and tied weights (W_d = W_eᵀ) produces which similarity structure?

Why is a two-tower model with one-hot inputs memory-based, while an autoencoder is model-based?

The VAE reparametrisation trick e = μ + σ ⊙ ε is needed because:

12 · Recap

One-screen summary

Chapter 11 — load-bearing ideas

Layers → AE → CF: fully-connected layer $x_o=f(x_i W + b)$ ; autoencoder encodes $r_u\to e_u\to\hat r_u$ ; a shallow linear AE gives $S=W_eW_d$ (asymmetric) or $S=W_eW_e^\top$ (symmetric) — MF / item-CF as special cases.
Two-tower = non-linear MF: one-hot + linear = MF with factors $W^{(U)}_u\cdot W^{(I)}_i$ ; full-profile input = model-based (Asymmetric SVD). Autoencoders are always model-based.
The zoo: DAE corrupts the input (salt & pepper); VAE encodes a distribution with the reparametrisation trick; EASE-R is a closed-form linear AE with $\operatorname{diag}(S)=0$ — fast and effective, but memory-intensive.
Always compare to simple baselines: depth and non-linearity often add little on the sparse CF benchmark — the reproducibility warning.