Language Models & Embeddings · Natural Language Processing

01 · Definition

What is a language model?

A language model assigns a probability to a sequence of words — equivalently, it predicts the next word given the previous ones. That single capability underlies autocomplete, machine translation, speech recognition, and every modern LLM.

By the chain rule of probability, any sequence factorises exactly into a product of conditionals:

Chain rule

P(w_1, \dots, w_n) = \prod_{i=1}^{n} P(w_i \mid w_1, \dots, w_{i-1})

Exact, but useless as written — we can never estimate $P(w_i \mid w_1,\dots,w_{i-1})$ for long histories, because that exact history has almost never been seen.

Hands-on

Chain rule of probability

Any sentence probability factorises exactly into a product of conditionals, P(w₁…wₙ) = ∏ P(wᵢ | w₁…wᵢ₋₁). The n-gram model then truncates each context to the previous n−1 words.

Sentence

P(“the cat sat on the mat”) =

P(the | ⟨s⟩) = 0.050

× P(cat | the) = 0.402

× P(sat | the cat) = 0.315

× P(on | cat sat) = 0.276

× P(the | sat on) = 0.213

× P(mat | on the) = 0.180

= 6.68e-5(log₂ = -13.87)

TakeawaySentence probabilities get vanishingly small (note the exponent), which is why language models always work in log-space — products of probabilities become sums of log-probabilities.

02 · The Markov assumption

n-grams

The fix is the Markov assumption: condition only on the previous $n-1$ words. A bigram model uses one word of history, a trigram two:

n-gram

P(w_i \mid w_1, \dots, w_{i-1}) \approx P(w_i \mid w_{i-n+1}, \dots, w_{i-1})

Estimate each conditional by counting: $P(w_i \mid w_{i-1}) = \dfrac{\operatorname{count}(w_{i-1}, w_i)}{\operatorname{count}(w_{i-1})}$ (maximum-likelihood estimate).

Hands-on

n-gram next-word predictor

An n-gram model predicts the next word from the previous n−1. More context sharpens the prediction but hits unseen contexts sooner — the sparsity that smoothing fixes.

Corpus (one document per line)

Context

34 tokens · 14 types · 12 contexts

context: "the" · total 4

film

2/4 = 0.500

acting

2/4 = 0.500

TakeawayEach step up in n means more context but exponentially more (and sparser) contexts to estimate. The Markov assumption (condition on the last n−1 words) is what makes n-grams tractable.

Q

Why higher n is not always better

Larger $n$ captures more context but the number of possible contexts grows as $|V|^{n-1}$ , so counts become sparse — most $n$ -grams are never seen, and their MLE probability is zero. The bias–variance trade-off in disguise: low $n$ underfits (too little context), high $n$ overfits (memorises the training corpus, fails on unseen contexts).

03 · Unseen events

Smoothing

MLE assigns probability zero to any n-gram not seen in training — fatal, since a single zero makes the whole sentence probability zero. Add-α (Laplace) smoothing moves a little mass to every event:

Add-α

P(w \mid c) = \frac{\operatorname{count}(w, c) + \alpha}{\operatorname{count}(c) + \alpha\,|V|}

Hands-on

Add-α (Laplace) smoothing

At α = 0 (MLE), three unseen words have probability exactly zero — fatal for a language model. Raise α and mass flows to them, keeping the distribution normalised.

α0.00

0.500

cat

0.375

dog

0.125

bird

0.000

fish

0.000

car

0.000

sun

P(w) = (c + α) / (N + α·V), with N = 8, V = 6. · 3 words have probability zero (MLE).

TakeawaySmoothing trades a little probability away from seen events to guarantee no event is impossible. Add-1 is crude; Kneser–Ney and back-off / interpolation are the production-grade versions.

key

Beyond add-1: back-off and interpolation

Add-1 over-smooths in practice. Back-off uses the trigram if it was seen, else falls back to the bigram, else the unigram. Interpolation always mixes all three with learned weights. Kneser–Ney smoothing — the production standard for n-grams — estimates how likely a word is to appear in a novel context, not just how frequent it is.

04 · Generation

Generating text

A language model also generates: sample the next word from $P(\cdot \mid \text{context})$ , append it, repeat. The decoding strategy controls the trade-off between coherence and diversity:

Greedy / argmax

Always take the highest-probability word. Deterministic and locally optimal, but repetitive and bland — and not globally optimal.

Temperature sampling

Divide logits by $T$ before the softmax. $T < 1$ sharpens (safer, more repetitive); $T > 1$ flattens (more diverse, more errors); $T \to 0$ recovers greedy.

Top-k

Sample only from the $k$ most probable words, renormalised. Cuts off the unreliable tail.

Top-p (nucleus)

Sample from the smallest set whose cumulative probability exceeds $p$ . Adapts the candidate set to how peaked the distribution is.

Q

Temperature, precisely

Temperature $T$ rescales logits $z_i \to z_i / T$ before softmax. Lower $T$ concentrates probability on the top words (greedy-like, deterministic); higher $T$ spreads it out (creative, riskier). It is the single most common knob for trading coherence against diversity.

05 · Evaluation

Perplexity

The intrinsic metric for a language model is perplexity — the exponential of the per-word cross-entropy. Intuitively, the model’s average branching factor: how many words it is, on average, choosing between. Lower is better.

Perplexity

\text{PP}(W) = 2^{H(W)}, \qquad H(W) = -\frac{1}{N}\sum_{i=1}^{N} \log_2 P(w_i \mid w_{<i})

Hands-on

Perplexity calculator

Perplexity is the model’s average branching factor — lower is better. A good model assigns high probability to the actual next word; a uniform model is maximally surprised.

Sentence

step	P(w \| prev)	−log₂
P(the \| <s>)	0.9000	0.15
P(cat \| the)	0.4000	1.32
P(sat \| cat)	0.5000	1.00
P(on \| sat)	0.9000	0.15
P(the \| on)	0.9500	0.07
P(mat \| the)	0.1500	2.74

cross-entropy H = 0.91 bits/word → perplexity = 1.87

TakeawayPerplexity = 2^H, where H is cross-entropy in bits/word. It is the standard intrinsic metric for language models; an unseen or low-probability word spikes it dramatically.

Q

Interpreting perplexity

A uniform model over $|V|$ words has perplexity $|V|$ (maximally uncertain). A perfect model that always assigns probability 1 to the actual next word has perplexity 1. One unseen word (probability ≈ 0) sends $-\log_2 P$ huge and spikes perplexity — which is why smoothing matters for evaluation too.

06 · The wall

Why n-grams fail

Two structural problems no amount of smoothing fixes:

Sparsity. Even a trigram model over a 50 k vocabulary has $10^{14}$ possible contexts; almost all are unseen.
No notion of similarity. “the cat sat” and “the dog sat” are independent events to an n-gram — it learns nothing about one from the other. Words are atomic symbols with no shared structure.

The second problem is the deeper one. We need a representation where similar words are close, so that evidence transfers between them. That is the entire motivation for embeddings.

07 · Distributional semantics

The distributional hypothesis

“You shall know a word by the company it keeps.” — J.R. Firth, 1957

Words that appear in similar contexts tend to have similar meanings. Make it operational: represent each word by the words it co-occurs with. The co-occurrence matrix counts, for each word, how often every other word appears within a context window — each row is a (sparse, high-dimensional) vector for that word.

Hands-on

Co-occurrence matrix

Count how often each word appears within ±window of each other word. Words with similar rows share contexts — the distributional hypothesis: “you shall know a word by the company it keeps.”

Corpus

Window ±2

	and	the	film	was	brilliant	moving	a	acting	boring	dull
and	0	1	2	4	2	2	4	0	2	2
the	1	0	4	4	0	1	0	2	0	0
film	2	4	0	2	1	1	0	2	1	0
was	4	4	2	0	1	0	0	2	2	0
brilliant	2	0	1	1	0	2	1	0	0	0
moving	2	1	1	0	2	0	1	0	0	0
a	4	0	0	0	1	1	0	0	0	2
acting	0	2	2	2	0	0	0	0	1	0
boring	2	0	1	2	0	0	0	1	0	1
dull	2	0	0	0	0	0	2	0	1	2

TakeawayCo-occurrence counts are the bridge from symbols to vectors: a word’s row is a (sparse, high-dimensional) context vector. Word2Vec and GloVe learn dense versions of exactly this.

key

From counts to dense vectors

Co-occurrence rows are huge and sparse. Reduce them (SVD → LSA, or learn them directly → word2vec / GloVe) into dense vectors of a few hundred dimensions where geometric distance reflects semantic similarity. That is a word embedding.

08 · Neural LMs

Neural language models

Instead of counting, a neural network learns the conditional distribution. Map each context word to its embedding, combine them (concatenate or average), pass through hidden layers, and softmax over the vocabulary to predict the next word. Two wins over n-grams: embeddings let the model generalise across similar words (evidence for “cat” helps “dog”), and a fixed-size hidden state can in principle carry longer context than any fixed $n$ . The recurrent and transformer architectures (Ch. 5–6) are elaborations of this idea.

09 · word2vec

word2vec — embeddings as a by-product of prediction

word2vec trains a shallow network on a self-supervised task and keeps the embeddings. Skip-gram predicts the context words from a centre word; CBOW predicts the centre word from its context. Negative sampling makes training cheap: instead of a full softmax over $|V|$ , distinguish real (word, context) pairs from a few random “negative” ones. No labels are needed — the text is its own supervision.

Hands-on 1

Analogies are vector arithmetic

Pick an analogy. The widget takes the relation vector from b → a and applies the same arrow starting at c — the head lands on the answer word. The two arrows are parallel because the "male → female" and "country → capital" offsets are roughly constant directions in the space.

king − man + woman ≈ queen ✓

Try thisSwitch between king − man + woman and paris − france + italy. The two arrows in each plot are the same length and direction — that parallel structure is exactly what "analogy" means geometrically.

TakeawayThe regularity (constant "male→female" / "country→capital" offsets) emerges from the context-prediction objective — no loss term ever asked for it. And because embeddings are real vectors, adding and subtracting them is perfectly meaningful.

Q

Skip-gram vs CBOW, and negative sampling

Skip-gram (centre → context) works better for rare words and small corpora; CBOW (context → centre) is faster and smooths over frequent words. Negative sampling replaces the expensive $|V|$ -way softmax with a handful of binary “is this a real context word?” decisions — the key efficiency trick.

10 · Geometry of meaning

Properties of the embedding space

The striking result is that linear structure emerges without ever being asked for:

Similarity — nearest neighbours by cosine are semantically related (cat near dog, kitten, pet).
Analogy — constant offset directions: $\text{king} - \text{man} + \text{woman} \approx \text{queen}$ , $\text{paris} - \text{france} + \text{italy} \approx \text{rome}$ (the widget above).
Bias — the same geometry encodes social biases present in the training corpus (e.g. occupation–gender associations). Embeddings inherit the data’s prejudices; debiasing is an active concern.

!

Static embeddings have one vector per word

word2vec/GloVe give one vector per word type, so “river bank” and “savings bank” collapse to the same vector — they cannot disambiguate sense. Contextual embeddings (ELMo, BERT — Ch. 6) fix this by producing a different vector per occurrence, conditioned on the sentence.

11 · Applications

Where embeddings are used

Pre-trained embeddings became the default input layer for nearly every NLP model of the late 2010s: initialise a classifier, tagger, or NER model with word2vec/GloVe vectors and it trains faster and generalises better, especially with limited labelled data. They also power semantic search (embed query and documents, rank by cosine — the dense counterpart to Ch. 3’s TF-IDF) and are the conceptual seed of the contextual representations that follow.

12 · Self-check

Questions before you move on

What does the Markov assumption let an n-gram model do?

Why does maximum-likelihood (un-smoothed) estimation fail for a language model?

A model assigns the actual next word probability 1 at every step. Its perplexity is:

What is the key limitation of n-grams that dense embeddings solve?

13 · Recap

One-screen summary

Chapter 04 — load-bearing ideas

A language model assigns probability to word sequences via the chain rule, $P(w_{1:n}) = \prod_i P(w_i \mid w_{<i})$ .
n-grams apply the Markov assumption — condition on the previous n−1 words — and estimate conditionals by counting.
Smoothing (add-α, back-off, interpolation, Kneser–Ney) gives unseen events non-zero probability; MLE alone zeroes whole sentences.
Generation = sampling the next word; greedy / temperature / top-k / top-p trade coherence against diversity.
Perplexity = $2^{H}$ , the per-word cross-entropy exponentiated — the model’s average branching factor (lower is better).
n-grams fail on sparsity and on having no notion of word similarity.
Distributional semantics → word2vec: words are known by their company; learned dense embeddings put similar words close, and analogies appear as constant offset directions.

14 · Exam · past papers

Past-paper questions

Answered 0 / 8 · 0 correct

Q-LM1A language model is best described as a model that:
Q-LM2The bigram approximation P(wᵢ | w₁…wᵢ₋₁) ≈ P(wᵢ | wᵢ₋₁) is justified by:
Q-LM3Why is Laplace (add-1) smoothing applied to n-gram probabilities?
Q-LM4Perplexity of a language model is:
Q-LM5The distributional hypothesis states that:
Q-LM6Which is TRUE of word2vec embeddings?
Q-LM7The analogy "king − man + woman ≈ queen" demonstrates that embedding space:
Q-LM8Lowering the softmax temperature T during generation: