Chapter 12

Factorization Machines

MF sees only users and items; linear regression sees any feature but only linearly. Factorization Machines do both: they accept arbitrary one-hot feature vectors (users, items, genres, context) and learn factorised pairwise interactions between every feature pair. Instead of a dense O(n²) weight matrix W, FM sets W ≈ VVᵀ — one latent vector per feature — unifying global effects, MF, hybrid, and context-aware recommenders in a single equation.

Reading: ~58 min Interactive: 1 widgets Source: Polimi Recommender Systems 2024/25 — Factorization Machines · Rendle, Factorization Machines (ICDM 2010)

key

The big idea

Start with a linear model on one-hot features $\hat r = \omega_0 + \sum_i \omega_i x_i$ — global effects and per-feature biases. Add all pairwise feature interactions $\sum_i\sum_{j>i}\omega_{ij}\,x_i x_j$ — but a full weight matrix $W=(\omega_{ij})$ is $O(n^2)$ , far too large for sparse data. The FM insight: factorise $W$ , exactly as Chapter 8 factorised the URM. Each feature $i$ gets a latent vector $v_i\in\mathbb{R}^f$ and $\omega_{ij}=v_i\cdot v_j$ . Parameters drop from $O(n^2)$ to $O(nf)$ — and the model now generalises across feature pairs, because every feature shares one latent vector in all its interactions.

01 · One feature vector per interaction

From rating matrix to feature matrix

FM does not work on the URM directly. It reshapes every (user, item, rating) triple into a single feature vector.

The FM input is a feature matrix where each row is one user–item interaction $k=(u,i)$ and each column is a binary feature. For a pure collaborative setting the columns are the one-hot encoding of both the $\lvert U\rvert$ users and the $\lvert I\rvert$ items:

FM input vector (collaborative)

x^{(k)} = [\,\underbrace{0,\dots,1,\dots,0}_{U\text{ bits}},\;\underbrace{0,\dots,1,\dots,0}_{I\text{ bits}}\,]\;\in\;\lbrace 0,1\rbrace^{U+I}.

The row for user $u$ rating item $i$ has exactly two 1’s — at position $u$ and at position $U+i$ — and zeros elsewhere. This is the same information as the URM, arranged differently: FM sees one row per observed rating, the URM one row per user. The target for row $k$ is the rating $r^{(k)}=r_{ui}$ — so FM is a supervised regression on $x^{(k)}$ .

key

Why one-hot?

One-hot encoding makes every feature independent. FM sees “user=Bob” and “item=Interstellar” as separate columns and learns a latent vector for each. When we later add content or context features, they are simply more one-hot columns appended to the same row — no algorithm change. This is the key to FM’s generality: any categorical feature becomes a column.

02 · Constant + linear + interactions

The FM prediction equation

Three terms: a global bias, per-feature weights, and factorised pairwise feature interactions.

FM prediction

\hat r^{(k)} = \omega_0 + \sum_{i=1}^{n}\omega_i\,x_i^{(k)} + \sum_{i=1}^{n}\sum_{j=i+1}^{n}\omega_{ij}\,x_i^{(k)}x_j^{(k)}.

The three components:

Constant $\omega_0\in\mathbb{R}$ — the global intercept; estimate only this and you get the average rating across all interactions. The same role as $\mu$ in global effects (Ch. 2).
Linear $\sum_i \omega_i x_i^{(k)}$ — each feature gets a weight for its independent contribution. For collaborative one-hot input the only active features are user $u$ and item $i$ , so $\sum_i\omega_i x_i = \omega_u + \omega_i$ — exactly $b_u + b_i$ from Ch. 2. The linear term is global effects.
Quadratic $\sum_i\sum_{j>i}\omega_{ij}\,x_i x_j$ — pairwise interactions; the product $x_i x_j=1$ only when both features are active. This is what lifts FM beyond linear regression: it can learn that “user=Bob” and “item=Interstellar” interact positively (Bob likes sci-fi) while “user=Dan” interacts negatively.

key

The interaction term is the secret sauce

Without it, FM is just global effects — it cannot personalise. The interaction between user and item features is what makes FM a collaborative model. And because the interactions are factorised (next section), FM generalises to unseen $(u,i)$ pairs, exactly as MF does.

03 · The core contribution

Factorising the interaction matrix

A full pairwise interaction matrix has O(n²) parameters. Factorising it reduces to O(nf) and enables generalisation.

In vector form, $\hat r^{(k)} = \omega_0 + \boldsymbol{\omega}\cdot x^{(k)} + x^{(k)\top} W\,x^{(k)}$ with $W\in\mathbb{R}^{n\times n}$ symmetric. The problem: $W$ has $(n^2-n)/2$ free parameters — with $n=U+I$ features, far more than the number of observed ratings. We cannot learn this from sparse data. The fix is to factorise $W$ exactly as Ch. 8 factorised the URM:

FM factorisation

W \approx V V^\top,\qquad V\in\mathbb{R}^{n\times f},\; f \ll n,\qquad \omega_{ij} = v_i\cdot v_j = \sum_{h=1}^{f} v_{ih}\,v_{jh}.

Each feature $i$ now has a latent vector $v_i\in\mathbb{R}^f$ (row $i$ of $V$ ), and the interaction weight is their dot product. Parameters drop from $1 + n + \tfrac{n^2-n}{2}$ to $1 + n + nf$ — from quadratic to linear in $n$ . For 100 users and 1000 items ( $n=1100$ ), the unfactorised $W$ needs about 600k parameters; the factorised version with $f=10$ needs only 12,101.

Without factorisation

Each feature pair has its own weight

\omega_{ij}

, learned independently. No generalisation: if no one rated item X on a weekend, FM cannot estimate X’s weekend bias.

With factorisation

Features share latent vectors. Bob’s vector

v_{\text{Bob}}

contributes to every interaction involving Bob (items, genres, days…). Two features are similar if their latent vectors are close.

key

This is MF, applied to features

Ch. 8 factorised the rating matrix $R\approx X Y^\top$ into user and item embedding tables. FM factorises the feature interaction matrix $W\approx VV^\top$ into a single feature embedding table $V$ — every feature, whatever its type, shares one latent space. That shared space is what makes FM a unified model.

The final FM with factorised interactions:

FM final

\hat r^{(k)} = \omega_0 + \sum_{i=1}^{n}\omega_i\,x_i^{(k)} + \sum_{i=1}^{n}\sum_{j=i+1}^{n}\Big(\sum_{h=1}^{f} v_{ih}\,v_{jh}\Big) x_i^{(k)}x_j^{(k)}.

Hands-on

FM prediction · constant + linear + interaction

A collaborative FM on the shared URM. The prediction splits into ω₀ (global), ω_u+ω_i (the linear global-effects term), and v_u·v_i (the factorised interaction — the MF dot product). The interaction is what moves the score off a per-user constant.

Latent factors f2

ω₀ constant+1.468

ω_u+ω_i linear+1.698

v_u·v_i interact+0.837

prediction+4.003

actual rating+4.000

Bob · Interstellar — f=2: interaction v_u·v_i = 0.837 · observed 4

TakeawayDrop the interaction term and the FM is just global effects — every item rated by a user scores the same. The factorised v_u·v_i is the personalising piece, and because every feature shares one latent vector, FM generalises to unseen pairs and to new feature types (genres, context) added as columns.

04 · One model, many special cases

FM as a unifying model

Depending on which features you include, FM reproduces global effects, MF, SVD++, hybrid, and context-aware recommenders.

The beauty of FM is that different recommendation models are just different choices of input features fed into the same quadratic equation:

Global effects (Ch. 2): user and item one-hot features, but drop the quadratic term. Then $\hat r^{(k)}=\omega_0+\omega_u+\omega_i$ .
Matrix factorization (Ch. 8): user and item one-hot, keep the quadratic. The interaction $v_u\cdot v_i$ is the dot product of latent factors — exactly $x_u\cdot y_i$ . The lecture confirms an FM with only collaborative data is equivalent to SVD++.
Hybrid (Ch. 9): add content features (genres, actors) as extra one-hot columns. The quadratic term now includes user–genre and item–actor interactions — no model change.
Context-aware: add context features (day, time, location). The quadratic term learns user–context and item–context interactions.

Collaborative FM

Features:

U+I

columns. Quadratic term captures

v_u\cdot v_i

. Equivalent to SVD++.

Collaborative + content

Features:

U+I+\text{Actors}+\text{Genres}

. Quadratic captures user–genre, item–actor, actor–genre interactions in one objective.

Context-aware

Features:

U+I+\text{Day}+\text{Time}

. Quadratic captures user–weekend, item–evening interactions. Same FM, different columns.

key

The key difference from S-SLIM (Ch. 9)

S-SLIM extends SLIM by adding the ICM as pseudo-users in a stacked matrix. FM extends any feature set by factorising pairwise interactions. FM is more general — it handles categorical features (day, time, location) that don’t fit a user–item matrix — but both rely on factorisation to beat sparsity in the interactions.

05 · The imbalance problem

Factorization Machines with implicit feedback

With implicit data, the model must see negatives. Without them, it learns to predict 1 for everything.

A dataset with implicit ratings (interacted / not, 1/0) contains only positives in raw form. Train FM on positives alone and it learns to predict 1 for every $(u,i)$ — useless for ranking. The fix is the same as for BPR (Ch. 7) and BCE (Ch. 11): negative sampling. For each positive $(u,i)\in R^{+}$ , randomly draw an equal number of non-interacted items for $u$ and add them as negatives (target 0), balancing the training set:

FM implicit sampling

\mathcal{D} = \lbrace (x^{(k)}, 1)\,:\, r_{ui}=1\rbrace \;\cup\; \lbrace (x^{(k)}, 0)\,:\, \text{sampled non-rated }(u,i)\rbrace.

Sampling matters

If you draw negatives uniformly from the vast pool of non-interactions, the model sees far more negatives than positives and learns to predict 0 for everything. Balance the two sets (50/50 is common) — identical to the BCE sampling strategy of Ch. 11. The same principle holds regardless of model.

06 · Exam intel

What the exam tests

Write the FM equation with all three terms; explain the one-hot input ( $U+I$ bits, two 1’s per row); show the linear term equals global effects; explain why $W\approx VV^\top$ reduces parameters from $O(n^2)$ to $O(nf)$ ; derive $\omega_{ij}=v_i\cdot v_j$ ; name at least three models FM subsumes; and describe the implicit-feedback sampling strategy.

Worked question — count the parameters

A collaborative FM has $\lvert U\rvert=100$ users and $\lvert I\rvert=500$ items. (a) Length $n$ of $x^{(k)}$ ? (b) With $f=10$ , how many parameters? (c) Without factorisation, how many would the quadratic term alone need? (d) One model FM reduces to with only user and item features?

(a) $n = U + I = 600$ . Each row has exactly two 1’s (one user, one item).
(b) $1 + n + nf = 1 + 600 + 600\times 10 = 6{,}601$ .
(c) $\tfrac{n^2-n}{2} = \tfrac{360{,}000-600}{2} = 179{,}700$ — far more than the observed ratings on sparse data. FM avoids this explosion.
(d) SVD++ / matrix factorization: $\omega_u + \omega_i + v_u\cdot v_i$ with the global bias.

Traps: counting columns as $U\times I$ instead of $U+I$ (it’s one row per rating, $U+I$ features); saying FM factorises the rating matrix (it factorises the feature interaction matrix $W$ , not the URM); forgetting the linear term is global effects; and confusing FM with S-SLIM (which stacks matrices, item-based only).

07 · Exam · past papers

Past-paper questions

Past paper Exam 2020 · CF recommender with FMs (6 pts)

Q. Implement a CF recommender with FMs: the input data structure (1); the prediction equation and its parameters (1); how parameters are learnt (1); the analogy with global effects + MF for explicit ratings (2); how FMs do context-aware recommendation (1).

Model answer. Input. One sparse feature vector $x$ per interaction: one-hot user block ⊕ one-hot item block ⊕ optional context/side-info blocks. Model. $\hat y(x)=w_0+\sum_i w_i x_i+\sum_{i<j}\langle v_i,v_j\rangle x_i x_j$ ; parameters are the global bias $w_0$ , per-feature weights $w_i$ , and a $k$ -dim factor $v_i$ per feature. Learning. Minimise a loss (MSE for explicit, BCE/BPR for implicit) by SGD or ALS with L2 regularization. Analogy. With only user/item one-hots, $w_0+w_u+w_i$ reproduces global effects and $\langle v_u,v_i\rangle$ the MF dot product — so FM = global effects + MF. Context-aware. Append context blocks (time, device, location); the pairwise term automatically models user×context and item×context through the shared factors.

Past paper RS Exam · FM data structure and terms (7 pts)

Q. The data structure used by FMs and its relation to the URM (2); write the FM model (2); explain the terms and their relation to other models (2); how to use FMs for context-aware recommendation (1).

Model answer. Data ↔ URM. A flat design matrix: each observed URM entry $(u,i)$ becomes one training row $x$ with its user and item one-hot blocks active (label = rating). The URM is “unrolled” into feature vectors, which lets extra columns (context, attributes) be appended. Model. $\hat y(x)=w_0+\sum_i w_i x_i+\sum_{i<j}\langle v_i,v_j\rangle x_i x_j$ . Terms. $w_0$ = global average; $w_i$ = linear per-feature bias ( $\approx$ global effects); $\langle v_i,v_j\rangle$ = factorised pairwise interaction ( $\approx$ MF / SVD++). Context-aware. Add context blocks; their factors interact with user/item factors automatically.

Past paper FT-Sample · FM ↔ MF + global effects (6 pts)

Q. Input data structure (1); prediction equation + parameters (1); how parameters are learned (1); the FM ↔ MF + global effects relationship for explicit ratings (2); context-aware use (1).

Model answer. Identical in substance to the 2020 question: input = sparse one-hot user ⊕ item ⊕ context; model $\hat y(x)=w_0+\sum_i w_i x_i+\sum_{i<j}\langle v_i,v_j\rangle x_i x_j$ ; learned by SGD/ALS on a regularised loss; the linear terms are global effects, the pairwise factor term is MF; context is handled by appending feature blocks.

Past paper Practice Exam 1 · Input structure & side information (5 pts)

Q. The input data structure, including how categorical features are handled (2); the FM model in summation notation with each component explained (1); how FMs incorporate side information and context (2).

Model answer. Input / categoricals. Each interaction is a sparse vector; categorical variables (user id, item id, genre, device…) are one-hot encoded into dedicated blocks; numeric features stay as single columns. Model. $\hat y(x)=w_0+\sum_i w_i x_i+\sum_{i<j}\langle v_i,v_j\rangle x_i x_j$ — a global bias, linear per-feature effects, factorised pairwise interactions. Side info / context. Append more one-hot/numeric blocks (attributes, time, location); the factorised term learns interactions between these and the user/item without adding per-pair parameters.

Past paper Practice Exam 3 · Global effects & implicit imbalance (5 pts)

Q. How global effects are incorporated into FM (2); the FM ↔ MF + global effects relationship for explicit ratings (1); how the imbalance problem is addressed with implicit ratings (2).

Model answer. Global effects. They are exactly the linear part: $w_0$ (global mean) plus the active $w_i$ for the user and item one-hots give the global + user + item biases. FM ↔ MF + GE. Linear terms = global effects; pairwise factor term $\langle v_u,v_i\rangle$ = MF — so an FM on user/item one-hots equals global effects + MF. Imbalance. Implicit data is overwhelmingly negative, so training would be swamped by zeros; FMs build a balanced positive/negative set — sampling a comparable number of unobserved pairs per positive — so the optimiser sees real signal.

Past paper Practice Exam 5 · Factorised interactions & SVD++ (5 pts)

Q. How FMs factorise the interaction term to cut parameters (2); the FM ↔ SVD++ equivalence with only collaborative data (1); how implicit ratings are handled with balanced sampling (2).

Model answer. Factorised interactions. A naive model needs a free $w_{ij}$ per pair ( $O(n^2)$ , and untrainable for never-co-observed pairs); FMs replace it with $\langle v_i,v_j\rangle$ using $k$ -dim factors ( $O(nk)$ ), letting interactions generalise across pairs via shared factors — and the pairwise sum even computes in linear time. FM = SVD++. With user, item, and the user’s implicit-feedback features, an FM reproduces SVD++ — the factor interactions recover both the latent dot product and the implicit-feedback term. Balanced sampling. Sample a balanced set of negatives per positive so the loss is not dominated by the majority class.

08 · Self-check

Three questions before you move on

Why does FM factorise the interaction weight matrix W ≈ VVᵀ instead of learning W directly?

A collaborative-only FM with one-hot user and item features, keeping the quadratic term, is equivalent to which classic model?

How does FM achieve hybrid recommendation compared to S-SLIM (Ch. 9)?

09 · Recap

One-screen summary

Chapter 12 — load-bearing ideas

FM = linear + factorised interactions: $\hat r = \omega_0 + \sum\omega_i x_i + \sum_{i<j}(v_i\cdot v_j)x_i x_j$ . Constant = global bias, linear = per-feature weights (global effects), quadratic = pairwise interactions factorised via latent feature vectors.
Factorise $W$ , not $R$ : $W\approx VV^\top$ turns $O(n^2)$ into $1+n+nf$ parameters. Each feature shares its vector across all interactions — that is how FM generalises to unseen pairs.
One model, many inputs: collaborative features → MF/SVD++; add content columns → hybrid; add context columns → context-aware. The FM equation never changes, only the feature vector does — a unified framework.
Implicit feedback needs balanced negative sampling, or the model predicts 1 (or 0) everywhere.