Seq2Seq, Attention & Transformers · Natural Language Processing

01 · Encoder–decoder

Sequence-to-sequence and its bottleneck

Translation, summarisation, and dialogue map one sequence to another of different length. The seq2seq architecture uses an encoder RNN to read the source into a context vector and a decoder RNN to generate the target from it.

The flaw: the entire source must be compressed into a single fixed-size vector. For long sentences, information is lost before the decoder ever starts — BLEU collapses with length.

Hands-on

The seq2seq bottleneck

A vanilla encoder squeezes the whole source sentence into one fixed-size context vectorc, which the decoder reads. Drag the source length and watch capacity run out.

Source length6

Source information to encode (grows linearly with length):

Context vector c (fixed size, regardless of input length):

Comfortable. The context vector summarises this sentence with low loss — BLEU stays high.

TakeawayOne fixed vector cannot hold an arbitrarily long sentence. Attention lets the decoder read all encoder states directly — and dropping recurrence entirely gives the Transformer.

02 · The fix

Attention

Attention removes the bottleneck: instead of one context vector, the decoder, at each step, computes a weighted sum of all encoder states, where the weights say how relevant each source token is right now. The learned alignment is soft — a target word can attend partly to several source words.

Hands-on

Encoder–decoder attention heatmap

Each target word (row) attends over the source words (columns); weights sum to ~1. The bright off-diagonal cells are real soft alignments — reordering, fusion, and article shifts a fixed context vector could never capture.

	the	cat	sat	on	the	mat
le	0.78				0.10
chat		0.82
était			0.62			0.24
assis			0.74			0.12
sur				0.82
le					0.81
tapis						0.82

TakeawayAttention removes the seq2seq bottleneck: instead of cramming the whole source into one fixed vector, the decoder looks back at every source token at every step, weighting them by relevance.

key

From additive to scaled dot-product attention

Early (Bahdanau) attention scored alignments with a small neural net. The Transformer uses scaled dot-product attention: score = $\mathbf{q} \cdot \mathbf{k} / \sqrt{d_k}$ , softmax, weighted sum of values. The $\sqrt{d_k}$ keeps dot products from growing with dimension and saturating the softmax.

03 · The core operation

Self-attention — query, key, value

The Transformer’s insight: drop recurrence entirely and let every token attend to every other token in the same sequence. Each token is projected into three vectors — a query (what am I looking for?), a key (what do I offer?), and a value (what do I contribute?):

Scaled dot-product attention

\operatorname{Attention}(Q, K, V) = \operatorname{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right) V

Every query scores against every key; the softmax turns scores into attention weights; the output is the weighted sum of values. Fully parallel across positions — no sequential dependency, unlike an RNN.

Hands-on

Self-attention — query, key, value

Pick a query token. It scores against every key (qᵀk / √d), softmaxes into attention weights, and its new representation is Σᵢ αᵢ·vᵢ. The ambiguous “bank” attends to “money” in one sentence and “river” in the other — that is context disambiguation, for free.

Financial context:

River context:

Query = bank · scoring against every key:

token

qᵀk/√d

weight

α

I

0.00

0.14

withdrew

0.32

0.19

money

0.35

0.19

from

0.06

0.14

the

0.00

0.14

bank (self)

0.40

0.20

TakeawayAttention output(token) = Σᵢ αᵢ·vᵢ — a soft, content-based mixture of every value vector, weighted by how much the query “asked about” each key. No recurrence, fully parallel, and every token can reach every other in one step.

Q

Why self-attention beats recurrence

Parallelism — all positions computed at once, not left-to-right. Path length — any two tokens interact in one step, so long-range dependencies are as easy as short ones (vs an RNN’s $O(n)$ path). The cost is $O(n^2)$ attention in sequence length — the motivation for the efficiency tricks in Ch. 9.

04 · The architecture

The Transformer block

A Transformer layer stacks a few sub-components, each wrapped in a residual connection and layer normalisation:

Multi-head attention

Run $h$ attention “heads” in parallel, each with its own Q/K/V projections, then concatenate. Different heads learn different relations (syntax, coreference, position).

Feed-forward network

A position-wise two-layer MLP applied to each token independently — where much of the model’s capacity (and parameters) live.

Residual + LayerNorm

Add the input back (gradient highway) and normalise. These are what make very deep stacks trainable.

Stack ×N

Repeat the block N times (12 for BERT-base, 96+ for the largest models). Depth builds abstraction.

05 · Inputs

Subword tokenisation, revisited

Transformers operate on subword tokens (BPE / WordPiece / SentencePiece — Ch. 1): a fixed vocabulary of ~30–50 k pieces, no [UNK], morphology shared across related words. Each token id indexes an embedding matrix; the position signal is added on top. Special tokens ([CLS], [SEP], <bos>, <eos>) mark structure the model learns to use.

06 · Order

Positional encoding

Self-attention is permutation-invariant — shuffle the tokens and the output set is unchanged. So the model must be told the order. The original Transformer adds fixed sinusoidal position vectors to the token embeddings:

Sinusoidal PE

\text{PE}(p, 2i) = \sin\!\left(\frac{p}{10000^{2i/d}}\right), \qquad \text{PE}(p, 2i+1) = \cos\!\left(\frac{p}{10000^{2i/d}}\right)

Hands-on

Positional encoding

Self-attention is order-blind, so transformers add a position signal to each token embedding. Each row is a sinusoid of a different frequency; together they encode position uniquely. Indigo = positive, red = negative; brightness = magnitude.

Positions32Dimensions32

32 positions × 32 dimensions · PE(pos, 2i) = sin(pos / 10000^(2i/d)), PE(pos, 2i+1) = cos(…)

TakeawayFixed sinusoids let the model attend by relative position and extrapolate to longer sequences than seen in training. Modern models often use learned or rotary (RoPE) encodings instead — same goal, different parameterisation.

Q

Why sinusoids, and the modern alternatives

Each dimension is a sinusoid of a different wavelength, so positions get unique, smoothly-varying codes, and relative offsets become linear — the model can learn “attend 3 tokens back”. Sinusoids also extrapolate beyond training lengths. Modern models often prefer learned absolute or rotary (RoPE) encodings (Ch. 9).

07 · Two families

BERT vs GPT — encoders and decoders

The same block, two masking regimes, two families:

Encoder (BERT)

Bidirectional — every token attends to all others. Trained with masked language modelling (predict randomly masked tokens). Great for understanding tasks: classification, NER, QA. Not a generator.

Decoder (GPT)

Causal — a token attends only to earlier tokens (a triangular mask). Trained to predict the next token. Naturally generative; the architecture behind modern LLMs.

Encoder–decoder (T5, BART)

An encoder reads the input; a decoder generates the output attending to both its own prefix and the encoder. Best for seq2seq tasks (translation, summarisation).

Q

The masking is the difference

BERT’s bidirectional attention sees the whole sentence, so it cannot be used to generate left-to-right (it would peek at the answer). GPT’s causal mask hides the future, making next-token prediction well-defined. Same Transformer block — the attention mask determines whether you get an understander or a generator.

08 · Transfer learning

Pretrain, then fine-tune

The recipe that made Transformers dominate: pretrain a large model on huge unlabelled text with a self-supervised objective (masked- or next-token prediction), learning general language structure. Then fine-tune on a small labelled dataset for a specific task — usually just adding a task head and continuing training at a low learning rate. Pretraining is expensive and done once; fine-tuning is cheap and done per task. Lightweight variants (adapters, LoRA — Ch. 10) tune only a few parameters.

Q

Why pretraining transfers

Self-supervised pretraining learns syntax, semantics, and world knowledge from raw text — representations that transfer to almost any downstream task. Fine-tuning then needs far less labelled data than training from scratch, and generalises better. This is transfer learning, and it is why a single pretrained model seeds an entire ecosystem.

09 · Self-check

Questions before you move on

What problem does attention solve in the seq2seq architecture?

In scaled dot-product attention, what is the query used for?

Why do Transformers need positional encodings?

The key architectural difference between BERT and GPT is:

10 · Recap

One-screen summary

Chapter 06 — load-bearing ideas

Seq2seq encoder–decoder maps sequences of different lengths but bottlenecks the source into one fixed vector.
Attention fixes it: the decoder attends to all encoder states with learned, soft, relevance weights.
Self-attention = scaled dot-product over query/key/value; every token attends to every token, fully in parallel.
The Transformer block = multi-head attention + position-wise feed-forward, each wrapped in residual + layer norm, stacked N times.
Positional encoding restores word order to a permutation-invariant model (sinusoidal, learned, or rotary).
BERT (bidirectional encoder, masked LM) understands; GPT (causal decoder, next-token) generates; encoder–decoder (T5/BART) does seq2seq.
Pretrain → fine-tune is the transfer-learning recipe: expensive self-supervised pretraining once, cheap task fine-tuning many times.

11 · Exam · past papers

Past-paper questions

Answered 0 / 8 · 0 correct

Q-TR1The bottleneck of a vanilla (no-attention) seq2seq model is that:
Q-TR2In scaled dot-product attention, why divide the scores by √dₖ?
Q-TR3Self-attention computes, for each token, an output equal to:
Q-TR4What is the main advantage of self-attention over recurrence?
Q-TR5Multi-head attention is used because:
Q-TR6Positional encodings are necessary because self-attention is:
Q-TR7BERT is pretrained with which objective?
Q-TR8GPT-style decoders use a causal (triangular) attention mask so that: