Chapter 05

Word Embeddings

By the end you can explain how an autoencoder learns a representation with no labels, why one-hot words are useless for meaning, what task word2vec actually trains on (self-supervised but NOT an autoencoder), and where its analogy arithmetic comes from.

Reading: ~50 min Interactive: 1 widgets Source: Mikolov et al. (2013) word2vec · Pennington et al. (2014) GloVe · Bengio et al. (2003) NNLM

01 · Neural autoencoders

Learning a representation without labels

This chapter is about unsupervised representation learning — discovering good features from raw data with no labels. The clearest example is the autoencoder: a network trained to reconstruct its own input, with two halves and a narrow waist between them:

Autoencoder

\mathbf{x} \xrightarrow{\;\text{encoder } f\;} \mathbf{h} \xrightarrow{\;\text{decoder } g\;} \hat{\mathbf{x}}, \qquad \mathcal{L} = \lVert \mathbf{x} - g(f(\mathbf{x})) \rVert^2

The target is the input — no human labelling — which makes it self-supervised. The narrow code $\mathbf{h}$ is the representation we want; the decoder is often discarded afterwards.

To reconstruct $\mathbf{x}$ from the narrow code $\mathbf{h}$ , the encoder is forced to keep only the input’s salient structure and discard noise.

🗜️ Undercomplete

$\dim(h) \lt \dim(x)$ . A compression bottleneck — the standard case. Forces a compact code.

🕳️ Sparse / overcomplete

$\dim(h) \ge \dim(x)$ with a sparsity penalty so few units are active. Learns interpretable, parts-like features without a size bottleneck.

key

A linear autoencoder is PCA

An undercomplete autoencoder with linear activations and MSE loss learns the same subspace as PCA. The payoff of a neural autoencoder is the non-linearity: it can fold data onto a curved manifold that PCA’s straight axes can never capture — non-linear dimensionality reduction.

Why this matters — the applications recur throughout deep learning:

Dimensionality reduction / visualisation: compress to 2–3 dims, or to a compact code for downstream models.
Denoising autoencoders: corrupt the input, train to reconstruct the clean version. The network can’t just copy, so it learns robust structure.
Anomaly detection: train on normal data; anything with high reconstruction error is an outlier — exactly why anomaly detection is unsupervised (Chapter 01).
Variational Autoencoders (VAE): impose a probability distribution on the latent code so you can sample it and generate new data — the autoencoder turned generative.

The rest of this chapter applies the same “learn a dense representation unsupervised” idea to words — though, as we’ll see, word2vec reaches the goal by a different route than reconstruction.

02 · From one-hot to distributed

Words that know nothing about each other

The problem. Treating words as discrete symbols (one-hot vectors of size $\lvert V\rvert$ ) is high-dimensional, sparse, and semantically blind: every pair of words is orthogonal, so “speaks” is orthogonal to “addresses” and “Illinois” to “Chicago”. The model can’t generalise across words that mean the same thing.

why

A drawer with no cross-references

One-hot encoding files every word in its own drawer with no cross-references. “King” and “queen” are as unrelated as “king” and “carburettor”. A word embedding instead places words in a continuous space where nearby = similar in meaning — “you shall know a word by the company it keeps” (Firth, 1957).

A word embedding maps each word in a vocabulary $V$ (often above $10^6$ ) to a dense vector in $\mathbb{R}^m$ (typically 100–500). It fights the curse of dimensionality three ways: compression (dimensionality reduction), smoothing (discrete → continuous), and densification (sparse → dense).

The mapping

w \in V \;\xrightarrow{\;C\;}\; \mathbb{R}^m, \qquad \lvert V\rvert \gg m

Where embeddings came from: language modelling

The motivation was language modelling — predicting the next word from the previous ones. The classical approach is the N-gram model, which estimates probabilities by counting:

N-gram language model

P(w_t \mid w_1,\dots,w_{t-1}) \approx P(w_t \mid w_{t-n+1},\dots,w_{t-1})

N-grams have crippling limits: the number of possible contexts is $V^{n-1}$ (curse of dimensionality), most never appear in training (data sparsity → zero probabilities patched with ad-hoc smoothing), and — because words are one-hot — seeing “the cat sat” teaches the model nothing about “the dog sat”. Bengio’s Neural Net Language Model (2003) fixed this by learning a distributed word representation and the probability function jointly: similar words get similar vectors, so the model generalises to word sequences it never saw. The embeddings were a by-product — and word2vec is what happens when you optimise that by-product directly.

03 · Word2Vec

A shallow classifier, trained on a fake task

Bengio’s model was accurate but slow (a big hidden layer over a huge vocabulary). Word2Vec (Mikolov, 2013) made embeddings practical by stripping the model down — no hidden layer (~1000× speedup) and a shared projection — so it could train on billions of words. It comes in two flavours:

🎯 CBOW

Context → target. Predict the centre word from its surrounding window. Faster, good for frequent words.

🔄 Skip-gram

Target → context. Predict the surrounding words from the centre word. Better for rare words.

The crucial point for the exam: word2vec is a classifier over the vocabulary, trained with cross-entropy to predict a word from its context (or vice versa). The “labels” come for free from raw text, so it is self-supervised — but the embeddings are a by-product of that classification task, read off the projection (weight) matrix. (In practice the full softmax over a million words is replaced by negative sampling or hierarchical softmax to stay cheap.)

word2vec is NOT an autoencoder

An autoencoder reconstructs its input (output = input, trained by reconstruction error — §1). Word2vec’s input (context) and output (target word) are different, and it minimises a classification cross-entropy, not a reconstruction loss. Both are self-supervised, but word2vec is a predictive model, not an autoencoder — and its output is dense, not sparse (the smaller space comes from projection, not from a sparsity penalty).

Exam · 2026 Q3 — word2vec (every statement is false)

A classic all-false true/false. Why each is wrong:

✗ “Embeddings represent discrete words and can’t be added/subtracted” — they are dense vectors; analogy arithmetic works.
✗ “Trained unsupervised because it optimises vector distances, not a learning loss” — it minimises the cross-entropy of a context classifier.
✗ “A big improvement because it is sparse” — it is dense; the saving comes from a low-dimensional projection.
✗ “Not trained in an unsupervised manner” — it is self-supervised (labels come free from context).
✗ “Unsupervised because it is an autoencoder” — input ≠ output; it predicts context words, it does not reconstruct its input.

04 · Regularities, GloVe & applications

Why the analogies work — and what embeddings are for

The famous property: directions in the embedding space encode relationships, so analogies become vector arithmetic:

Analogy arithmetic

w_{king} - w_{man} + w_{woman} \approx w_{queen}, \qquad w_{paris} - w_{france} + w_{italy} \approx w_{rome}

Hands-on 1

Analogies are vector arithmetic

Pick an analogy. The widget takes the relation vector from b → a and applies the same arrow starting at c — the head lands on the answer word. The two arrows are parallel because the "male → female" and "country → capital" offsets are roughly constant directions in the space.

king − man + woman ≈ queen ✓

Try thisSwitch between king − man + woman and paris − france + italy. The two arrows in each plot are the same length and direction — that parallel structure is exactly what "analogy" means geometrically.

TakeawayThe regularity (constant "male→female" / "country→capital" offsets) emerges from the context-prediction objective — no loss term ever asked for it. And because embeddings are real vectors, adding and subtracting them is perfectly meaningful.

Emergent, not engineered

The constant “male→female” or “country→capital” difference vectors emerge from the context-prediction objective. They are not forced by any dedicated loss term — a favourite exam distractor. And because embeddings are real vectors, they certainly can be added and subtracted.

GloVe (Pennington, 2014) makes explicit what word2vec does implicitly: it factorises the global co-occurrence matrix, encoding meaning as ratios of co-occurrence probabilities, trained by weighted least squares:

GloVe objective

J = \sum_{i,j} f(X_{ij})\big(w_i^\top \tilde{w}_j + b_i + \tilde{b}_j - \log X_{ij}\big)^2

$X_{ij}$ counts how often word $j$ appears in the context of word $i$ ; $f$ down-weights very frequent pairs. word2vec is “predictive/local”; GloVe is “count-based/global” — but they learn similar geometry.

What embeddings are used for

🔎 Information retrieval

Represent a query and documents as embedding aggregates; rank by cosine similarity rather than exact word match.

📄 Document similarity

Average (or pool) word vectors into a document vector, then cluster or classify it.

💬 Sentiment analysis

Feed the embedding sequence into an LSTM (Chapter 04) — embeddings are the standard Embedding input layer.

That last use is the bridge to Chapter 04: a pre-trained embedding table turns each token into a dense vector before the recurrent network ever sees it, giving the sequence model a head start on meaning.

Exam · 2025 Q5 — word embedding true/false

The mixed key:

✓ Embedding transforms a one-hot word into a dense vector.
✓ Training is unsupervised (self-supervised — labels come free from context).
✗ “The embedding-space regularity is forced explicitly by a specific loss” — it emerges from the context-prediction objective.
✓ Skip-gram and CBOW are the two word2vec training schemes.

05 · Exam intel

What the exam actually tests

Word2vec true/false questions are the reliable points here — and they all hinge on the same handful of distinctions.

The word2vec distinction cluster

Word2vec is self-supervised (labels free from context), trained with cross-entropy as a vocabulary classifier, producing dense vectors. It is not an autoencoder (input ≠ output, no reconstruction loss), not sparse, and its analogy structure is emergent, not imposed by a loss.

Autoencoder facts

An autoencoder reconstructs its input (self-supervised); a linear autoencoder with MSE equals PCA; the non-linear version does manifold learning. Uses: dimensionality reduction, denoising, anomaly detection, and (as a VAE) generation.

CBOW vs skip-gram, word2vec vs GloVe

CBOW predicts the centre word from context (good for frequent words); skip-gram predicts context from the centre word (good for rare words). word2vec is predictive/local; GloVe factorises the global co-occurrence matrix — count-based/global. Similar geometry, different route.

06 · Common mistakes

Where students get this wrong

"word2vec is an autoencoder"

An autoencoder reconstructs its input; word2vec predicts a different word (context ↔ target) with a cross-entropy classifier. Same self-supervision, completely different model.

"word2vec is sparse / saves parameters via sparsity"

The output is dense. The win over one-hot comes from a low-dimensional projection, not from a sparsity penalty.

"The analogy structure is imposed by a special loss"

The constant relation vectors emerge from the context-prediction objective. No loss term ever asks for “male→female” to be constant.

"Discrete word vectors can't be added or subtracted"

Embeddings are real-valued vectors in $\mathbb{R}^m$ — arithmetic is exactly what makes analogies work.

"word2vec is supervised because it minimises a loss"

Having a loss doesn’t make it supervised. The targets are generated from the text itself (the surrounding words), so it is self-supervised / unsupervised.

07 · Self-check

Can you answer these?

Four questions in the exact shapes the exam uses. Click an option for instant feedback.

Why is word2vec self-supervised but NOT an autoencoder?

The relation king − man + woman ≈ queen works because…

What is the core weakness of one-hot word vectors that embeddings fix?

In word2vec, which describes skip-gram (as opposed to CBOW)?

08 · Recap

One-screen summary

Chapter 05 — load-bearing ideas

An autoencoder learns a representation with no labels by reconstructing its input (self-supervised); the bottleneck code is the goal. Linear + MSE = PCA; non-linearity is the payoff. Uses: dim-reduction, denoising, anomaly detection, VAE generation.
One-hot encoding is sparse and semantically blind (all words orthogonal); embeddings give dense vectors where proximity = similarity. N-grams fail (sparsity, no cross-word generalisation); Bengio’s NNLM learned embeddings jointly to fix it.
word2vec is a shallow classifier over the vocabulary, trained with cross-entropy on a context-prediction “fake task” — self-supervised, but not an autoencoder, and dense not sparse. CBOW: context→target; skip-gram: target→context.
Analogy arithmetic (king − man + woman ≈ queen) emerges from the objective; it is not imposed by a special loss.
GloVe factorises the global co-occurrence matrix (predictive/local vs count-based/global). Embeddings power IR, document similarity/classification, and feed the LSTMs of Chapter 04.