Recurrent Neural Networks · Deep Learning

01 · Sequential data and memory

When order carries the information

The problem. A feedforward net takes a fixed-size input and has no memory. But text, audio and time series are variable-length and their meaning lives in the order. We need a model whose output at time $t$ can depend on what came before.

🧮 Memoryless models

Autoregressive models and FFNNs with “delay taps”: predict the next value from a fixed window of past ones. No internal state.

🧠 Models with memory

Linear dynamical systems (Kalman), Hidden Markov Models (Viterbi), and RNNs — they carry a hidden state forward in time.

A Recurrent Neural Network keeps a hidden state $h_t$ that mixes the current input with the previous state — a distributed, non-linear memory:

RNN recurrence

h_t = \sigma(W_{hh}\,h_{t-1} + W_{xh}\,x_t + b_h)

The same weight matrices $W_{hh}, W_{xh}$ are reused at every time step — weight sharing across time.

why

Reading a sentence

An RNN reads a sentence the way you do — one word at a time, updating a running “gist” (the hidden state). The gist is everything it remembers; nothing else from earlier words survives.

02 · BPTT & the vanishing gradient

Why a vanilla RNN forgets

Backpropagation Through Time (BPTT) unrolls the RNN for $U$ steps and back-propagates as if it were a deep feedforward net — but with one twist: every copy of the recurrent weights is the same matrix, so the replicas are kept equal and updated with the average of their per-step gradients.

The gradient that must travel from time $t$ back to time $k$ is a product of Jacobians:

Gradient through time

\frac{\partial h_t}{\partial h_k} = \prod_{i=k+1}^{t}\frac{\partial h_i}{\partial h_{i-1}}

Derivation The norm bound: why sigmoid/tanh guarantee vanishing

For the scalar case $h_t = h(v\,h_{t-1} + w\,x)$ , each factor is $v\cdot h'(\cdot)$ . Taking norms over $t-k$ steps:

\left\lVert\frac{\partial h_t}{\partial h_k}\right\rVert \le \left(\gamma_v\cdot\gamma_{h'}\right)^{t-k}

where $\gamma_v$ bounds the weight and $\gamma_{h'}$ bounds the activation slope. With sigmoid ( $h' \le \tfrac14$ ) or tanh ( $h' \le 1$ ), the product $\gamma_v\cdot\gamma_{h'}$ is typically below 1, so the bound decays exponentially to 0: the gradient vanishes and long-range dependencies cannot be learned. If the product exceeds 1 it instead explodes.

×

Cause vs symptom

The symptom is “no learning of long-range structure”. The cause is the repeated multiplication of factors below 1. A first fix forces the factors to 0/1 (ReLU); the deeper fix is to make the recurrence additive with $v = 1$ (“only accumulate”) — which is exactly what the LSTM cell does.

Q

Exam · 2024 Q2 — the vanishing gradient, four parts

A recurring essay. The full-marks answer:

(a) What is it? During backprop the gradient becomes vanishingly small in early layers / timesteps, so those weights stop learning.
(b) What causes it? Repeated multiplication of Jacobians/derivatives below 1 (sigmoid $\sigma' \le 0.25$ , tanh $' \le 1$ ) across many steps; aggravated by small initialisation.
(c) Which architectures? Deep feedforward nets and especially vanilla RNNs (the recurrence multiplies the same Jacobian every timestep).
(d) Which fixes? ReLU (derivative 0/1), careful init (Xavier/He), BatchNorm, residual/skip connections, and LSTM/GRU gating (the additive cell-state highway).

03 · LSTM & GRU

Gating builds a gradient highway

The LSTM (Hochreiter & Schmidhuber, 1997) adds a cell state $C_t$ updated additively and protected by three sigmoid gates:

🚪 Forget gate fₜ

How much of the previous cell state to keep. $f \approx 1$ = remember, $f \approx 0$ = erase.

📥 Input gate iₜ

How much of the new candidate $\tilde{C}_t$ to write into the cell.

📤 Output gate oₜ

How much of the cell to expose as the hidden state.

LSTM cell & hidden state

C_t = f_t\odot C_{t-1} + i_t\odot \tilde{C}_t, \qquad h_t = o_t\odot\tanh(C_t)

The key is that the cell update is additive. When $f_t \approx 1$ , $C_t \approx C_{t-1}$ , and $\partial C_t/\partial C_{t-1} = f_t$ — a single factor near 1, not a product of weight matrices. That is the “gradient highway” that defeats vanishing. Set the forget gate near 1 in the lab and watch the cell state persist:

Hands-on 1

The forget gate is the gradient highway

At t = 0 a memory is written into the cell state. The forget gate f decides how much survives each step: C_t = f·C_(t-1) + i·C̃_t. Push f toward 1 and the memory persists for the whole sequence; drop it and the cell forgets at once.

Forget f0.95

Input i1.00

Output o1.00

Retention f^9

0.63

∂C_t/∂C_0

f^t

Memory

Preserved

Try thisSet f = 1.00: the cell state is a flat line — perfect memory, and the gradient factor ∂C_t/∂C_0 = f^t stays 1, so nothing vanishes. Now drag f down to 0.5 and watch both the memory and the gradient highway collapse. The output gate o only rescales the dashed h_t — it never touches the cell's persistence.

TakeawayVanishing gradients come from multiplying many factors below 1. The LSTM's additive cell update with f ≈ 1 replaces that product with a single near-1 factor — a highway the gradient travels without decaying. The forget gate, not the output gate, is what defeats vanishing.

×

Trap (2026 Q5)

It is the additive cell-state update (governed by the forget gate) that prevents vanishing gradients — not the output gate. The output gate only controls what is exposed as $h_t$ .

The GRU (Cho, 2014) simplifies the LSTM: it merges the forget and input gates into a single update gate $z_t$ and merges the cell and hidden state, adding a reset gate $r_t$ that controls how much past state feeds the candidate:

GRU

h_t = (1-z_t)\odot h_{t-1} + z_t\odot \tilde{h}_t, \qquad \tilde{h}_t = \tanh\!\big(W[r_t\odot h_{t-1},\,x_t]\big)

The same convex-combination structure ( $1-z$ and $z$ ) keeps an additive path for the gradient, so a GRU also defeats vanishing — with fewer parameters (two gates, no separate cell) and usually comparable accuracy. Rule of thumb: try a GRU first; reach for an LSTM if you need the extra capacity. In code, a stacked bidirectional LSTM is two lines:

import tensorflow as tf
tfkl = tf.keras.layers

model = tf.keras.Sequential([
    tfkl.Input((None,)),
    tfkl.Embedding(vocab, 128),
    tfkl.Bidirectional(tfkl.LSTM(64, return_sequences=True)),  # stacked...
    tfkl.Bidirectional(tfkl.LSTM(64)),                         # ...bidirectional
    tfkl.Dense(1, activation='sigmoid'),  # sentiment head
])

Q

Exam · 2026 Q5 — the six LSTM equations, one sentence each

Given the LSTM diagram and its six equations, name them top-to-bottom:

Input gate $i_t = \sigma(W_i[h_{t-1},x_t]+b_i)$ — how much new candidate information to write.
Candidate $\tilde{C}_t = \tanh(W_C[h_{t-1},x_t]+b_C)$ — the proposed new cell content.
Forget gate $f_t = \sigma(W_f[h_{t-1},x_t]+b_f)$ — how much of the previous cell state to keep.
Cell update $C_t = f_t\odot C_{t-1} + i_t\odot\tilde{C}_t$ — the additive memory highway (the key to avoiding vanishing gradients).
Output gate $o_t = \sigma(W_o[h_{t-1},x_t]+b_o)$ — how much of the cell to expose.
Hidden state $h_t = o_t\odot\tanh(C_t)$ — the emitted output at time $t$ .

LSTM = Long Short-Term Memory. The marks hinge on attributing the highway to equation (4), not the output gate.

04 · Architectures & sequence shapes

Stacked, bidirectional, and the order question

Two structural extensions, plus a property that trips up many exam answers:

Stacked (hierarchical) LSTMs feed one recurrent layer’s outputs into the next, building higher-level temporal features.
Bidirectional LSTMs run one pass left-to-right and one right-to-left, concatenating the hidden states — usable only when the whole sequence is available (not for streaming / autoregressive generation).
The initial state can be a learned parameter rather than zeros.

×

LSTMs are NOT permutation invariant

An LSTM’s state depends on the order of its inputs — it is the opposite of permutation invariant. To encode an unordered (multi)set identically every time you must first sort the items (2026 Q4). “Same set → same encoding because the LSTM is permutation invariant” is false.

Match the wiring to the task shape: one-to-many (image captioning), many-to-one (sentiment), many-to-many (translation), or synced many-to-many (frame labelling).

Q

Exam · 2026 Q4 — which sequence designs are sound?

Judge whether each design and its stated assumption make sense:

✓ Multiset encoder that sorts items first, then feeds them to an LSTM → consistent, order-independent encoding (sorting is what restores permutation invariance).
✓ Bidirectional LSTM spellchecker (left + right context) and a stacked Bi-LSTM sentiment classifier with a sigmoid — both valid.
✗ “Feed the set unsorted because the LSTM is permutation invariant” — an LSTM is order-dependent; different orders give different states.
✗ “We cannot reverse a sentence with an autoregressive LSTM” — it can perfectly well emit a reversed sequence.

Q

Exam · 2024 Q4 — the "Clever Hans" tap generator

Generate taps until a visual cue (a change in someone’s posture/face) says stop. The sound choices:

✓ An RNN over a stream of images (a conv front-end is likely, since the input is visual), emitting a per-step binary decision <tap> / <eos>, trained with binary cross-entropy.
✗ A CNN that regresses a tap count up front — throws away the sequential, cue-terminated nature.
✗ A text-to-sequence seq2seq — the trigger is the visual cue, not the question text.

The shape is autoregressive generation conditioned on a visual stream — recurrent + conv, per-step stop decision.

05 · Seq2seq & decoding

Encoder–decoder, teacher forcing, beam search

For variable-length input → variable-length output (translation), the encoder compresses the source into a context vector and the decoder generates the target autoregressively. This is a conditional language model:

Conditional language model

P(y_1,\dots,y_n\mid x) = \prod_{t=1}^{n} p(y_t\mid y_{\lt t},\,x)

The encoder–decoder template, by varying what the encoder consumes and what the decoder emits, covers a whole family of tasks — each a different shape of sequence problem:

🖼️ Image captioning

One-to-many. A CNN encodes the image into the context vector; an RNN decoder emits a word sequence.

💬 Sentiment analysis

Many-to-one. An RNN reads the sentence; the final hidden state feeds a classifier. No decoder needed.

🌐 Machine translation

Many-to-many. Encoder reads the source, decoder generates the target — the canonical seq2seq.

Practical training and decoding details the exams probe:

Teacher forcing: at training time feed the ground-truth previous token to the decoder; at inference feed the model’s own previous output (exposure bias).
Special tokens: <SOS> starts the decoder, <EOS> ends a sequence, <PAD> equalises batch lengths, <UNK> replaces rare words.
Greedy decoding takes the per-step argmax (cannot backtrack); beam search keeps the $K$ best partial hypotheses and usually wins.

Greedy vs the true objective

y' = \arg\max_y \prod_t P(y_t\mid y_{\lt t},x) \;\approx\; \prod_t \arg\max_{y_t} P(y_t\mid y_{\lt t},x)

Greedy replaces the global argmax over whole sequences with a per-step argmax — fast, but it can paint itself into a corner. Beam search hedges with $K$ candidates.

Q

Exam · 2025 Q3 / 2024 Q3 — attention & Transformers (outlook)

Beyond the core, but two past questions test the ideas:

Attention vs self-attention: attention (seq2seq) aligns the decoder state to the encoder states; self-attention relates every token of one sequence to every other (Q, K, V from the same input).
Scoring functions: dot-product has no learned parameters; Luong (general) adds one weight matrix; Bahdanau (additive) adds a small MLP.
Cross attention: in multi-head attention you can “cross” Q from the decoder with K/V from the encoder — that is encoder–decoder attention.
Transformers (true): residual connections in every block limit vanishing, so very deep Transformers train; the forward pass is deterministic. (False): “must have both encoder and decoder” — BERT is encoder-only, GPT decoder-only; and positional encodings are not always sinusoidal-and-added (some are learned).

06 · Exam intel

What the exam actually tests

RNNs are exam-dense — vanishing gradients and LSTM gating appear repeatedly, and sequence-design judgement is a favourite.

Q1

Vanishing gradient: cause vs symptom

Always separate the symptom (early layers / timesteps stop learning) from the cause (repeated multiplication of Jacobian factors below 1). List the fixes: ReLU, good init, BatchNorm, residual/skip connections, and LSTM/GRU gating.

Q2

Why the LSTM works — name the highway

The additive cell update $C_t = f_t\odot C_{t-1} + i_t\odot\tilde{C}_t$ gives $\partial C_t/\partial C_{t-1} = f_t$ . With $f_t \approx 1$ the gradient flows undecayed. Credit the forget gate / additive cell, never the output gate.

Q3

Match the architecture to the task

Order-dependent (so sort to encode a set); bidirectional needs the whole sequence (no streaming); seq2seq for variable→variable; many-to-one for classification; one-to-many for captioning. Justify with the shape of input and output.

07 · Common mistakes

Where students get this wrong

×

Confusing the symptom of vanishing gradients with the cause

“No long-range learning” is the symptom. The cause is multiplying many Jacobian factors below 1 across time. Fixing it means breaking that product — additive cell states, ReLU, skip connections.

×

Crediting the output gate for the gradient highway

The highway is the additive cell update with the forget gate near 1 ( $\partial C_t/\partial C_{t-1} = f_t$ ). The output gate only decides how much of the cell is exposed as $h_t$ .

×

"LSTMs are permutation invariant"

They are explicitly order-dependent — different input orders give different states. To encode an unordered set identically you must sort the items first.

×

Using a bidirectional RNN for streaming generation

A bidirectional pass needs the whole sequence up front. It cannot be used for autoregressive / online generation, where future tokens aren’t available yet.

×

"Greedy decoding is optimal"

Greedy takes the per-step argmax and cannot backtrack, so it can miss the globally best sequence. Beam search keeps $K$ hypotheses and usually decodes better.

08 · Self-check

Can you answer these?

Four questions in the exact shapes the exam uses. Click an option for instant feedback.

Why does a vanilla RNN fail to learn long-range dependencies?

What specifically lets an LSTM carry gradients across many time steps without vanishing?

You want to encode an unordered set of items so the same set always maps to the same vector, using an LSTM. What must you do?

At inference, greedy decoding takes the highest-probability token at each step. Why might beam search do better?

09 · Recap

One-screen summary

Chapter 04 — load-bearing ideas

RNNs carry a hidden state; vanilla RNNs forget because BPTT multiplies many Jacobians below 1 (vanishing) — separate cause from symptom.
The LSTM cell update is additive ( $C_t = f_t\odot C_{t-1} + i_t\odot\tilde{C}_t$ ); $\partial C_t/\partial C_{t-1} = f_t$ is the gradient highway. The forget gate — not the output gate — defeats vanishing.
LSTMs are order-dependent, never permutation invariant; sort inputs to encode a set identically.
Bidirectional needs the whole sequence; it cannot be used for autoregressive generation.
Seq2seq = a conditional language model; teacher forcing trains it, beam search decodes it better than greedy.
GRUs merge the gates (update + reset) for fewer parameters and comparable accuracy — try a GRU first.