Using Pretrained Models · Natural Language Processing

01 · Recap

Three architectures, three jobs

The pretrained-transformer toolkit splits by architecture. Choosing the right one is half the battle.

Encoder (BERT)

Bidirectional, masked-LM pretrained. Best for understanding — classification, NER, extractive QA, embeddings.

Decoder (GPT)

Causal, next-token pretrained. Best for generation and few-shot prompting — the LLM family.

Encoder–decoder (T5, BART)

Reads then generates. Best for seq2seq — translation, summarisation.

02 · What they know

Probing pretrained representations

A probe is a small classifier trained on frozen representations to test what information they encode. Probing BERT shows that lower layers capture surface and syntactic features (POS, parse depth) while higher layers capture semantics and task-specific signal — the model learned a linguistic hierarchy no one designed. Probing keeps the transformer frozen, so a high probe accuracy means the information is already there, not learned by the probe.

03 · Adapting BERT

Fine-tuning BERT for understanding

To use BERT on a task, add a small head on top of the [CLS] representation (or per-token outputs for tagging) and continue training end-to-end at a low learning rate. With a good pretrained model, a few thousand labelled examples often suffice — the heavy lifting was done in pretraining.

Q

Feature extraction vs fine-tuning

Feature extraction freezes BERT and trains only the head (fast, cheap, weaker). Fine-tuning updates all weights (slower, needs more data and care, usually best). Between them sit parameter-efficient methods (adapters, LoRA — Ch. 10) that tune a tiny fraction of weights.

04 · Sentence embeddings

Sentence embeddings (SBERT)

A plain BERT [CLS] vector is a poor sentence embedding for similarity. Sentence-BERT fine-tunes BERT with a siamese objective so that cosine similarity between sentence vectors reflects semantic similarity. The result: encode each sentence once into a fixed vector, then compare millions of pairs by cheap cosine — exactly the IR machinery of Chapter 3, now over dense, meaning-aware vectors instead of sparse TF-IDF.

Hands-on

Cosine similarity — angle, not magnitude

Type a query; documents are ranked by cos(query, doc). Cosine ignores length and measures direction. Switch between TF-IDF and raw counts and watch the ranking shift.

Documents (one per line)

Query

D10.684

the film was brilliant and moving

D20.648

a brilliant and clever film

D50.391

a dull and predictable film

D30.242

the acting was great and moving

D40.142

the film was boring and dull

D60.000

the acting was boring and weak

TakeawayCosine = dot product of unit-normalised vectors. It is length-invariant, which is exactly what you want when ranking documents of wildly different sizes against a short query.

Q

Why SBERT, not cross-encoder BERT, for search

A cross-encoder (feed both sentences through BERT together) is most accurate but must re-run the model for every pair — $O(n^2)$ , hopeless at scale. SBERT (a bi-encoder) embeds each sentence independently once, so retrieval is a fast nearest-neighbour search. Accuracy for speed — often a cross-encoder then reranks SBERT’s top candidates (the two-stage pattern from Ch. 3).

05 · Retrieval at scale

Vector databases & semantic search

Storing millions of embeddings and finding nearest neighbours fast needs an approximate nearest neighbour (ANN) index — HNSW graphs, IVF, product quantisation — the engine of a vector database (FAISS, Pinecone, pgvector). Semantic search embeds the query, retrieves the closest document vectors by cosine, and returns them by meaning rather than exact keywords. This is the retrieval half of RAG (Ch. 10) and the dense counterpart to the inverted index.

06 · Multimodal

CLIP — text and images in one space

CLIP trains an image encoder and a text encoder jointly with a contrastive objective: matching (image, caption) pairs are pulled together, mismatched pairs pushed apart, in a shared embedding space. The payoff is zero-shot image classification — embed the image and a set of templated textual class prompts (a photo of a <label>) and pick the nearest — and text-to-image retrieval. The same “align two modalities by contrastive learning” recipe underlies much of modern multimodal NLP.

07 · The generation side

Generation models

Decoder LLMs are adapted differently. Instruction fine-tuning trains on (instruction, response) pairs so the model follows commands; RLHF (Ch. 8) aligns it to human preferences. But often no training is needed at all — a large enough model can be steered purely by the prompt, which is the topic of the next two sections.

08 · Decoding

Decoding strategies

A generator outputs a probability distribution over the next token; the decoding strategy turns that into text. Greedy/beam maximise likelihood (coherent but bland and repetitive); temperature + top-k / top-p sampling trade coherence for diversity.

Hands-on

Decoding strategies

Greedy takes the top word every step (deterministic, repetitive). Top-k / top-p sample from the trimmed head; temperature reshapes the distribution first. Press Generate to resample.

Temp1.0

Generated: the cat quickly walked

Step 1: chose the from the 0.50a 0.25my 0.15

Step 2: chose cat from cat 0.40dog 0.30robot 0.15

Step 3: chose quickly from quickly 0.35slowly 0.25happily 0.20

Step 4: chose walked from walked 0.40ran 0.30jumped 0.18

TakeawayThere is no single “right” decoding. Greedy/beam maximise likelihood but go bland and repetitive; sampling (with temperature + top-k/top-p) trades a little coherence for diversity. The knobs are task-dependent.

09 · Prompting

In-context learning

The headline capability of large decoders: they learn a task from demonstrations in the prompt, with no weight updates. Zero-shot gives only an instruction; few-shot prepends a handful of worked examples; the model continues the pattern.

Hands-on

In-context learning — zero / one / few-shot

Pick a task and a shot count. The prompt below is exactly what you send; the model continues the pattern. No fine-tuning — the “learning” happens entirely in the context window.

Task

Zero-shot (instruction only)

Classify the sentiment of the review as positive, negative, or neutral.
Review: This is the best book I have read all year.Sentiment: positive

Highlighted text = the continuation the model is expected to produce; everything above it is the prompt you actually send.

TakeawayIn-context learning lets one frozen model do many tasks with zero gradient updates — just instructions and a few demonstrations. More shots usually help, up to the context-window limit.

Q

In-context learning vs fine-tuning

Fine-tuning changes the model’s weights and needs a labelled dataset and training run. In-context learning changes nothing — the “training set” is a few examples placed in the context window at inference time. It is instant and flexible, but limited by context length and generally weaker than fine-tuning when plenty of labelled data exists.

10 · Self-check

Questions before you move on

What is a 'probe' in the context of pretrained models?

Why is SBERT (a bi-encoder) preferred over a cross-encoder for large-scale semantic search?

CLIP aligns images and text by:

Few-shot in-context learning differs from fine-tuning in that it:

11 · Recap

One-screen summary

Chapter 07 — load-bearing ideas

Match the architecture to the job — encoder (understand), decoder (generate), encoder–decoder (seq2seq).
Probing shows pretrained models encode a linguistic hierarchy; fine-tuning adapts them with little labelled data.
SBERT produces sentence embeddings whose cosine reflects meaning — dense semantic search.
Vector databases index embeddings for fast approximate nearest-neighbour retrieval (the dense inverted index).
CLIP aligns text and images in one space via contrastive learning, enabling zero-shot vision.
Decoding (greedy/beam vs temperature/top-k/top-p) trades coherence for diversity.
In-context learning (zero/one/few-shot) adapts a frozen model through the prompt — no weight updates.

12 · Exam · past papers

Past-paper questions

Answered 0 / 6 · 0 correct

Q-PT1You need to classify support tickets and have ~3,000 labelled examples. The most effective approach is usually to:
Q-PT2For fast semantic search over millions of documents you would use:
Q-PT3Probing a frozen BERT typically reveals that:
Q-PT4CLIP enables zero-shot image classification because it:
Q-PT5Greedy decoding tends to produce text that is:
Q-PT6Increasing the number of in-context examples (more shots) generally: