RAG, Agents & Adaptation
Extending a frozen LLM. Retrieval-augmented generation grounds answers in real documents; ReAct-style agents reason and call tools; LoRA fine-tunes cheaply; and distillation compresses a big model into a small one. The toolkit for shipping LLM applications.
01 · Grounding
Retrieval-augmented generation
An LLM’s knowledge is frozen at training time and prone to hallucination. Retrieval-augmented generation (RAG) fixes both: retrieve relevant documents at query time and put them in the prompt, so the model answers from sources rather than from memory.
The pipeline is exactly the search machinery of Chapters 3 and 7, plus a generator:
- Index — embed a document corpus into a vector store (offline).
- Retrieve — embed the query, find the nearest document chunks by cosine.
- Augment — paste those chunks into the prompt as context.
- Generate — the LLM answers grounded in the retrieved text, ideally with citations.
The retrieval step is the cosine ranking you already know — here over the toy corpus:
Cosine similarity — angle, not magnitude
Type a query; documents are ranked by cos(query, doc). Cosine ignores length and measures direction. Switch between TF-IDF and raw counts and watch the ranking shift.
Why RAG beats fine-tuning for knowledge
Fine-tuning bakes knowledge into weights — expensive to update, hard to attribute, and still hallucination-prone. RAG keeps knowledge external: update the index, not the model; cite the source; and the answer is grounded in retrieved text. Fine-tune to change behaviour/style; retrieve to supply facts. (This is exactly how this site’s AI tutor works — its index is built from the chapter prose.)
RAG quality = retrieval quality
A RAG system can only answer from what it retrieves. Garbage retrieval → garbage answer, however good the LLM. So chunking strategy, embedding model, and reranking (Ch. 3’s two-stage pattern) matter as much as the generator — and the index must actually contain the corpus text, which is why on-page examples and prose are written to be retrievable.
02 · Tool use
Agents and the ReAct loop
A bare LLM cannot look something up, run code, or take an action. An agent wraps the LLM in a loop that lets it use tools. The ReAct pattern interleaves Reason (think about what to do) and Act (call a tool), feeding the tool’s Observation back into the context:
Thought → Action (call a tool) → Observation → Thought → … → Answer
The model decides which tool to call (search, calculator, code, API), reads the result, and continues until it can answer. This turns a text predictor into something that can do things — at the cost of new failure modes (wrong tool, bad arguments, prompt injection from tool outputs, runaway loops).
Why agents need a loop, not one call
A single forward pass cannot observe the world. The ReAct loop lets the model gather evidence incrementally — call a tool, see the result, reason about it, call another — which is essential for multi-step tasks (look up X, compute Y from it, then answer). Guardrails (step limits, tool whitelists, output validation) are mandatory because each Observation is untrusted input.
03 · Cheap adaptation
LoRA — parameter-efficient fine-tuning
Full fine-tuning updates all of a model’s billions of weights — expensive, and you get a whole new copy per task. LoRA (Low-Rank Adaptation) freezes the base model and injects a small pair of low-rank matrices into each weight, training only those. The update has a tiny rank, so you train under 1% of the parameters, store a few-MB adapter per task, and swap adapters at serving time.
Why LoRA works
Fine-tuning updates tend to be low-rank — the change a task requires lives in a small subspace. LoRA parameterises exactly that subspace ( with small rank ), capturing most of full fine-tuning’s benefit at a fraction of the cost. Quantizing the frozen base and training LoRA on top (QLoRA) fine-tunes huge models on a single GPU.
04 · Compression
Knowledge distillation
To get a small, fast model that behaves like a big one, distill: train a small student to mimic the outputs (soft probabilities) of a large teacher. The teacher’s full distribution — including the relative probabilities of wrong answers — carries more signal than a hard label, so the student learns more from it than from the raw data alone. DistilBERT and many on-device models are distilled this way.
Distillation vs quantization vs pruning
Three ways to shrink a model, often combined. Distillation trains a smaller architecture to imitate a teacher (changes the model). Quantization (Ch. 9) lowers numeric precision (same architecture). Pruning removes low-importance weights/heads. Distillation gives the biggest size reduction but needs training; quantization is free at inference time.
05 · Self-check
Questions before you move on
What is the main benefit of RAG over relying on an LLM's parametric memory?
In the ReAct agent loop, what does the 'Observation' provide?
LoRA fine-tunes a model efficiently by:
Knowledge distillation trains a small student to:
06 · Recap
One-screen summary
Chapter 10 — load-bearing ideas
- RAG retrieves relevant documents at query time and grounds generation in them — external, updatable, citable knowledge that fights hallucination. RAG quality ≈ retrieval quality.
- The RAG pipeline is index → retrieve (cosine over embeddings) → augment the prompt → generate — the search machinery of Ch. 3/7 plus a generator.
- Agents (ReAct) loop Reason → Act (tool call) → Observe, turning a predictor into something that can act — with new failure modes to guard.
- LoRA adapts a frozen model by training tiny low-rank matrices (under 1% of params); QLoRA fine-tunes huge models on one GPU.
- Distillation trains a small student to mimic a big teacher’s soft outputs — the main way to get fast, small models; complements quantization and pruning.
07 · Exam · past papers
Past-paper questions
Q-RAG1The steps of a retrieval-augmented generation pipeline are, in order:
Q-RAG2A key advantage of RAG over fine-tuning for supplying factual knowledge is that:
Q-RAG3The ReAct agent pattern interleaves:
Q-RAG4LoRA reduces fine-tuning cost by:
Q-RAG5In knowledge distillation, the student benefits from the teacher’s soft labels because:
Q-RAG6A RAG system gives a wrong answer despite a capable LLM. The most likely first place to look is: