From Perceptrons to Neural Networks · Deep Learning

01 · The perceptron

From a biological neuron to a linear classifier

The whole field is a loose caricature of the brain. A biological neuron receives signals through its dendrites, accumulates them in the cell body (soma), and — if the accumulated charge crosses a threshold — fires a spike down its axon to other neurons across synapses. The synapses have variable strengths, and learning is the adjustment of those strengths. The artificial neuron keeps exactly this skeleton and throws away the biology:

🧠 Biological neuron

Dendrites collect inputs · synapse strengths weight them · the soma sums · it fires when a threshold is crossed · the axon carries the output onward.

⚙️ Artificial neuron

Inputs $x_i$ · weights $w_i$ (the learnable synapses) · a weighted sum · a threshold / activation · a single scalar output.

The historical lineage tracks this idea becoming trainable: McCulloch & Pitts (1943) proposed the Threshold Logic Unit (fixed weights, no learning); Hebb (1949) gave the first learning principle (“neurons that fire together wire together”); Rosenblatt (1957) built the trainable Perceptron; Widrow & Hoff (1960) added ADALINE with a bias term and a least-squares rule. A perceptron computes a weighted sum (the bias folds in as $w_0$ with $x_0 = 1$ ) and thresholds it:

Perceptron

h_j(\mathbf{x},\mathbf{w}) = h\!\left(\sum_{i=0}^{I}w_i x_i\right) = \mathrm{sign}(\mathbf{w}^\top\mathbf{x})

The decision boundary $w_0 + \mathbf{w}^\top\mathbf{x} = 0$ is a hyperplane. With the right weights a perceptron implements Boolean gates.

Worked example A perceptron that computes OR (and AND)

Take $w_0 = -\tfrac12,\; w_1 = 1,\; w_2 = 1$ . The unit fires when $-\tfrac12 + x_1 + x_2 > 0$ :

h_{\text{OR}}(x_1,x_2)=\mathbb{1}\!\left[-\tfrac12 + x_1 + x_2 > 0\right]

The input $(0,0)$ gives $-\tfrac12$ , which is below zero, so the output is 0; any input containing a 1 gives at least $\tfrac12$ , above zero, so the output is 1. That is OR. For AND, raise the threshold: $w_0 = -\tfrac32$ fires only when both inputs are 1. Each Boolean gate is just a different placement of the same separating line — which is exactly why the next problem is fatal.

×

What it cannot do: XOR

XOR is not linearly separable — no single line separates $\{(0,1),(1,0)\}$ from $\{(0,0),(1,1)\}$ . Minsky & Papert (1969) made this famous and triggered the first “AI winter”. The fix is depth: stack perceptrons so a hidden layer re-represents the inputs into a space where they are separable.

How does it learn? Hebbian / perceptron learning adjusts weights one sample at a time (online), only when a sample is misclassified:

Perceptron update

w_i^{k+1} = w_i^{k} + \eta\,x_i^{k}\,t^{k}\quad(\text{only on misclassified } k)

Deep dive Hebbian learning IS stochastic gradient descent

Code the outputs as $\pm 1$ . The error driving learning is the distance of the misclassified points from the boundary:

D(\mathbf{w},w_0) = -\sum_{i\in M} t_i\,(\mathbf{w}^\top\mathbf{x}_i + w_0)

This is non-negative and zero only when everything is correct. Its gradient with respect to $\mathbf{w}$ is $-\sum t_i\mathbf{x}_i$ , so an SGD step over one misclassified point is $\mathbf{w} \leftarrow \mathbf{w} + \eta\,t_i\mathbf{x}_i$ — exactly the Hebbian update. The perceptron convergence theorem then guarantees termination if and only if the data is linearly separable.

Hands-on 1

Make a perceptron compute OR, AND… and XOR

A perceptron fires when w₀ + w₁x₁ + w₂x₂ ≥ 0 — a single straight line. Tune the weights so the line separates the filled points (target 1) from the hollow ones (target 0). You can hit 4/4 for OR and AND. Try XOR — and watch why depth is needed.

w₁1.0

w₂1.0

w₀ (bias)-0.5

Correct

4 / 4

Gate

OR

Separable?

Yes

Try thisGet OR to 4/4 (e.g. w₁ = w₂ = 1, w₀ = −0.5), then switch the gate to XOR without touching the weights. No slider combination reaches 4/4 — the best any single line manages is 3/4.

TakeawayA perceptron draws one hyperplane, so it only solves linearly separable problems. XOR needs a hidden layer to re-represent the inputs into a space where they are separable — that is exactly what an MLP buys you.

02 · MLPs and backprop

Depth, universality, and the chain rule

A multi-layer perceptron (MLP) stacks non-linear layers; the hidden layers learn intermediate features that make the final decision linearly separable — this is how XOR gets solved. The universal approximation theorem (Hornik, 1991) — a single hidden layer of sigmoidal units can approximate any continuous function on a compact set — tells us one hidden layer is already enough, in principle, to represent any continuous mapping.

×

What the UAT does NOT promise

It guarantees a network exists — not that gradient descent will find it, not that the required width is feasible (it can be exponential), and not that the result will generalise. “Representable” is not “learnable”.

Backpropagation (Rumelhart, Hinton & Williams, 1986) computes gradients with the chain rule, in two passes: a forward pass computes activations; a backward pass propagates $\partial\mathcal{L}/\partial(\cdot)$ from output back to input. Each weight’s update is local:

Chain rule for a weight

\frac{\partial \mathcal{L}}{\partial w_{ji}} = \frac{\partial \mathcal{L}}{\partial g}\cdot\frac{\partial g}{\partial h_j}\cdot\frac{\partial h_j}{\partial w_{ji}}

Worked example Backprop through a 2-layer net (the w₃₅ example)

For a net $g(\mathbf{x}\mid\mathbf{w}) = g\!\big(\sum_j w_{1j}\,h_j(\sum_i w_{ji}x_i)\big)$ trained with sum-of-squared-errors $E = \sum_n (t_n - g)^2$ , the gradient with respect to a first-layer weight $w_{3,5}$ chains four local derivatives:

\frac{\partial E}{\partial w_{3,5}} = -2\sum_n (t_n - g)\,\underbrace{g'}_{\text{output}}\,\underbrace{w_{1,3}}_{\text{layer-2 weight}}\,\underbrace{h_3'}_{\text{hidden act.}}\,\underbrace{x_{5}}_{\text{input}}

Read it right-to-left: the input $x_5$ that fed the weight, the hidden unit’s slope $h_3'$ , the downstream weight $w_{1,3}$ carrying influence to the output, the output slope $g'$ , and the residual $(t-g)$ . Backprop simply reuses the shared front factors across all weights — that is why it costs two passes, not one pass per weight.

03 · Activation functions

Non-linearity, and the right output head

Without a non-linearity, stacked layers collapse to a single linear map — depth would buy nothing. And activations must be differentiable to train by gradient descent. Note each one’s derivative, because that factor appears in every backprop step:

📈 Sigmoid

$\sigma(a)=1/(1+e^{-a}) \in (0,1)$ . Derivative $\sigma' = \sigma(1-\sigma) \le \tfrac14$ → saturates, causing vanishing gradients.

〰️ Tanh

$\tanh(a) \in (-1,1)$ , with $\tanh' = 1-\tanh^2$ . Zero-centred (nicer than sigmoid) but still saturates.

📐 ReLU

$\max(0,a)$ . Derivative is 0 or 1 → no vanishing. Risk: dying neurons stuck at 0.

📊 Leaky ReLU

$\max(\alpha a, a)$ . A small negative slope keeps dead neurons alive.

The output head is chosen by the task and the label coding: linear for regression ( $\mathbb{R}$ ), tanh for two classes coded $\pm 1$ , sigmoid for $\{0,1\}$ (read as a posterior probability), and softmax for $K$ classes:

Softmax (K classes)

y_k = \frac{\exp(z_k)}{\sum_{k'}\exp(z_{k'})}

The outputs are positive and sum to 1 — a probability distribution over the $K$ classes.

Hands-on 2

Activations and their derivatives

The solid curve is the activation; the dashed curve is its derivative — the factor backprop multiplies at every layer. A derivative that flattens to zero (sigmoid, tanh) is what causes vanishing gradients in deep networks.

ReLU(x) = max(0, x)

f(0)

0

max f′

1

Saturates?

No (x>0)

Derivative is 0/1, so no vanishing for positive inputs — the default choice. Risk: a neuron stuck in the negative region never updates ("dying ReLU").

Try thisCompare the dashed derivative of Sigmoid (a low bump that dies to zero) with ReLU (a flat 1 for x > 0). Stacking many sigmoid layers multiplies those small numbers together — that is the vanishing gradient, and why ReLU became the default.

TakeawayPick an activation by the shape of its derivative: non-saturating (ReLU family) keeps gradients alive in deep nets; saturating (sigmoid/tanh) is fine shallow or at an output head, but chokes learning when stacked deep.

04 · Loss functions from MLE

Why MSE for regression, cross-entropy for classification

Loss functions are not arbitrary — they fall out of maximum likelihood estimation (MLE): choose the parameters that make the observed data most probable. The recipe is always the same: write the likelihood $L = P(\text{Data}\mid\theta)$ , take its log, differentiate, set to zero (or descend).

Worked example MLE warm-up: the mean of a Gaussian

For $x_1,\dots,x_N \sim \mathcal{N}(\mu, \sigma^2)$ , the log-likelihood is $\ell(\mu) = \text{const} - \tfrac{1}{2\sigma^2}\sum_n (x_n-\mu)^2$ . Setting $\partial\ell/\partial\mu = 0$ :

\frac{\partial \ell}{\partial \mu} = \frac{1}{\sigma^2}\sum_n (x_n-\mu) = 0 \;\Rightarrow\; \mu_{\text{MLE}} = \frac{1}{N}\sum_n x_n

MLE recovers the sample mean — a sanity check before we point it at a network.

Derivation Regression → MSE

Model the target as $t_n = g(x_n\mid\mathbf{w}) + \varepsilon$ , with $\varepsilon \sim \mathcal{N}(0,\sigma^2)$ . Then maximising the log-likelihood of the data is

\arg\max_{\mathbf{w}} \sum_n \log\frac{1}{\sqrt{2\pi}\,\sigma}\,e^{-\frac{(t_n-g(x_n\mid\mathbf{w}))^2}{2\sigma^2}} = \arg\min_{\mathbf{w}} \sum_n \bigl(t_n - g(x_n\mid\mathbf{w})\bigr)^2

The constants drop out; the Gaussian exponent is the sum of squared errors. Gaussian noise ⇒ MSE.

Derivation Binary classification → cross-entropy

Model the label as $t_n \sim \text{Bernoulli}\big(g(x_n\mid\mathbf{w})\big)$ . The likelihood is $g^{t}(1-g)^{1-t}$ ; its negative log over the data is

\mathcal{L} = -\sum_n\big[\,t_n\log g(x_n\mid\mathbf{w}) + (1-t_n)\log(1-g(x_n\mid\mathbf{w}))\,\big]

Bernoulli labels ⇒ binary cross-entropy. (Categorical labels ⇒ categorical cross-entropy with softmax.) The loss encodes the assumed noise model — that is the whole idea.

In code, the loss / output-activation pairing mirrors the MLE derivations exactly — linear + MSE for regression, softmax + categorical cross-entropy for $K$ -class classification:

import tensorflow as tf
tfkl = tf.keras.layers

# 2-2-1 net that can learn XOR (one hidden layer breaks linearity)
model = tf.keras.Sequential([
    tfkl.Input((2,)),
    tfkl.Dense(2, activation='tanh'),     # hidden layer
    tfkl.Dense(1, activation='sigmoid'),  # Bernoulli head
])
model.compile(loss='binary_crossentropy',  # = MLE under Bernoulli
              optimizer='adam', metrics=['accuracy'])

The whole story collapses into one table — memorise the column you need by remembering the noise model that generated it:

📈 Regression

Output: linear · Loss: SSE / MSE · Noise model: Gaussian.

⚖️ Binary classification

Output: sigmoid · Loss: binary cross-entropy · Noise model: Bernoulli.

🎲 Multi-class (K)

Output: softmax · Loss: categorical cross-entropy · Noise model: Categorical.

key

Why not just use SSE everywhere?

SSE on a sigmoid output is non-convex and gives near-zero gradients when the prediction is confidently wrong (the sigmoid saturates), so learning stalls. Cross-entropy is the MLE-correct loss for Bernoulli / Categorical labels and keeps the gradient healthy — another reason the loss must match the noise model, not personal taste.

Q

Exam · 2025 Q4 — which hold in the regression setting?

Each statement is independent; mark the true ones:

✓ L1 (sum of $\lvert w\rvert$ ) added to the loss — a valid regulariser.
✗ Categorical cross-entropy — that is a classification loss, not a regression loss.
✓ Normalise the target to $[-1,1]$ and use a tanh output — fine (it may saturate).
✓ Mean absolute error (MAE) — a standard regression loss.
✗ “MAE forces positive outputs” — the loss never constrains the output’s sign; positivity comes from the output activation (e.g. ReLU), not the loss.
✗ Multi-output regression with a softmax — softmax couples the outputs into one distribution; wrong for independent real values.
✓ Multi-output regression with ReLU — valid when the targets are non-negative.

The trap: softmax / cross-entropy belong to classification; in regression the activation, not the loss, controls the output range.

05 · Gradient descent variants

Batch, stochastic, and mini-batch

Training a network is non-convex optimisation; we descend the loss iteratively:

GD update

w^{k+1} = w^k - \eta\,\frac{\partial E(w)}{\partial w}\bigg|_{w^k}

$\eta$ is the learning rate — too small and convergence crawls; too large and the iterates overshoot.

📦 Batch GD

All $N$ samples per step. Exact gradient, but slow and memory-heavy.

🎲 Stochastic GD

One sample per step. Noisy / high-variance but fast; the noise can escape poor minima.

⚖️ Mini-batch GD

A batch of $M$ samples (32–256). The practical sweet spot — it maps cleanly onto GPU parallelism.

The tricks that speed up this descent — momentum, adaptive learning rates, careful initialisation, and BatchNorm — are the subject of Chapter 03.

06 · Exam intel

What the exam actually tests

This chapter is one of the most heavily examined — loss functions and the MLE story appear almost every year. Three shapes recur.

Q1

Derive a loss from its noise model

You’re given a task (“regression with Gaussian noise”, “binary labels”) and asked for the principled loss. Write the likelihood, take the negative log, and show the constants drop out: Gaussian → MSE, Bernoulli → binary cross-entropy, Categorical → categorical cross-entropy. One sentence earns the marks: “the loss is the negative log-likelihood of the assumed noise model.”

Q2

Why XOR needs depth — and what UAT really says

Explain that a perceptron is a single hyperplane, XOR is not linearly separable, so a hidden layer is required. If asked about the universal approximation theorem, state the catch: it proves existence, not learnability, feasible width, or generalisation.

Q3

Pick the output head + loss for a task

Given a task description, choose the output activation and loss together: linear + MSE (regression), sigmoid + binary cross-entropy (two classes), softmax + categorical cross-entropy ( $K$ classes). The sign / range of the output is set by the activation, never by the loss.

07 · Common mistakes

Where students get this wrong

×

"A perceptron can learn XOR with the right weights"

It cannot — XOR is not linearly separable, and one perceptron draws exactly one line. The best a single line achieves on XOR is 3 of 4 points. You need a hidden layer.

×

"UAT means one hidden layer is enough in practice"

The theorem promises a network exists, not that it is trainable, of feasible width, or able to generalise. Representable ≠ learnable. Depth is what makes the representation efficient and findable.

×

Using SSE for a classification head

SSE on a sigmoid is non-convex and its gradient vanishes when the model is confidently wrong, so training stalls. Cross-entropy is the MLE-correct loss for Bernoulli / Categorical labels and keeps gradients healthy.

×

Softmax for multi-output regression

Softmax couples the outputs into a single probability distribution that sums to 1 — wrong for predicting several independent real numbers. Use a linear head (or ReLU if the targets are non-negative).

×

"Backprop runs one gradient pass per weight"

Backprop is two passes total — one forward, one backward — regardless of how many weights there are. It reuses the shared front factors of the chain rule, which is exactly why it is cheap.

08 · Self-check

Can you answer these?

Four questions in the exact shapes the exam uses. Click an option for instant feedback.

Why can't a single perceptron compute XOR?

You model regression targets as t = g(x) + ε with ε Gaussian. Maximum likelihood gives which training loss?

For a K-class classification head trained by maximum likelihood, the correct output activation and loss are…

Which statement about ReLU versus sigmoid is correct?

09 · Recap

One-screen summary

Chapter 02 — load-bearing ideas

A perceptron is a linear classifier. Its learning rule is SGD on the misclassified-distance criterion, and it converges if and only if the data is linearly separable.
XOR is not linearly separable — depth solves it. The universal approximation theorem says one hidden layer can represent any function, not that we can train or generalise it.
Backprop = the chain rule in two passes (forward activations, backward gradients). Shared front factors make it cost two passes, not one per weight.
Losses come from MLE. Gaussian noise → MSE; Bernoulli labels → (binary) cross-entropy; Categorical → categorical cross-entropy. The loss encodes the assumed noise model.
The output head follows the task: linear (regression), sigmoid / tanh (binary), softmax ( $K$ -class). The output’s range is set by the activation, never by the loss.
Activations are chosen by their derivative. Non-saturating (ReLU family) keeps gradients alive deep; saturating (sigmoid/tanh) chokes learning when stacked.