Efficient Inference & Deployment · Natural Language Processing

01 · Deployment

Where a model runs

A trained model is only useful if you can serve it within a latency, cost, and memory budget. Inference, not training, dominates the lifetime cost of a deployed model — so the techniques in this chapter are about making a fixed model cheaper and faster to run, not more accurate.

The first decisions are where (cloud API, your own GPU, on-device) and what latency (interactive chat vs batch). These constrain everything downstream.

02 · Weights

Open vs closed weights

Closed (API)

You call a hosted model (GPT, Claude). No infra, always current — but per-token cost, data leaves your control, and you cannot fine-tune the base weights.

Open weights

You download the weights (Llama, Mistral, Qwen) and run them yourself. Full control, private, fine-tunable — but you own the GPUs, the serving stack, and the updates.

03 · Runtimes

Inference runtimes

Serving open weights efficiently is a software problem. Runtimes like vLLM, TensorRT-LLM, and llama.cpp add continuous batching, paged KV-cache memory, and kernel fusion to push throughput far above a naïve PyTorch loop. The same weights can run an order of magnitude faster (or slower) depending on the runtime and how it schedules requests.

04 · Precision

Quantization

Weights are usually trained in 16-bit float. Quantization stores them in fewer bits (INT8, INT4), cutting memory and bandwidth proportionally. Accuracy holds remarkably well down to ~4 bits, then falls off a cliff — which is why 4-bit is the popular sweet spot for running large models on consumer hardware.

Hands-on

Quantization — memory vs accuracy

Storing weights at fewer bits shrinks memory linearly but costs accuracy. Drag the bit-width and model size — watch where the accuracy cliff starts.

Precision

Params (B)7B

Weight memory

13.0 GB

Perplexity

5.51 (+0.1%)

TakeawayPost-training quantization to INT8/INT4 is the cheapest way to fit a big model on small hardware: memory drops with bit-width, accuracy holds down to ~4 bits, then falls off a cliff.

Q

Why quantization works (mostly)

Neural networks are robust to small weight perturbations, so rounding weights to a coarse grid changes outputs little — until the grid is too coarse (≤3 bits) and rounding error dominates. Post-training quantization is free; quant-aware training or methods like GPTQ/AWQ recover most of the remaining loss at low bit-widths.

05 · Caching

The KV cache

Generating token $t$ needs attention over all previous tokens’ keys and values. Without caching, you recompute them every step — $O(t)$ work per token, $O(n^2)$ total. The KV cache stores past keys and values so each new token only computes its own, turning generation into $O(n)$ — the single most important inference optimisation. Its cost is memory: the cache grows with sequence length × layers × heads, and often dominates GPU memory for long contexts.

Q

The KV-cache trade

The cache trades memory for compute: store every past key/value (large, grows with context) to avoid recomputing them (huge compute saving). Long-context serving is largely a fight to shrink this cache — which motivates grouped-query attention and sliding-window attention next.

06 · Attention memory

Grouped-query attention

The KV cache is dominated by having one key/value per attention head. Multi-query attention (MQA) shares a single key/value head across all query heads — tiny cache, slight quality loss. Grouped-query attention (GQA) is the middle ground: a few key/value heads shared among groups of query heads. GQA cuts KV-cache memory several-fold with almost no quality drop, and is standard in modern open models.

07 · Kernels

FlashAttention

Standard attention materialises the full $n \times n$ score matrix in slow GPU memory — memory-bandwidth bound, and quadratic in memory. FlashAttention computes attention in tiles that stay in fast on-chip SRAM, never writing the full matrix out. Same maths, exact result — just a memory-aware kernel that is several times faster and uses linear memory in sequence length, enabling much longer contexts.

08 · Positions

Rotary position embeddings (RoPE)

Instead of adding a position vector (Ch. 6), RoPE rotates the query and key vectors by an angle proportional to position. The dot product between a query at position $m$ and a key at position $n$ then depends only on the relative offset $m - n$ — relative position falls out for free, and the scheme extrapolates to longer sequences than seen in training (with interpolation tricks). RoPE is the default in most current open models.

09 · Sparse attention

Sliding-window attention

Full attention is $O(n^2)$ . Sliding-window attention restricts each token to attend only to the last $w$ tokens, making attention $O(n \cdot w)$ — linear in length. Stacked layers still propagate information globally (a token sees its window, whose tokens saw theirs), so long-range information flows across depth. A cheap way to extend context length.

10 · Stability

Pre-norm

Where the layer-normalisation sits matters for training deep stacks. Post-norm (the original Transformer) normalises after the residual add; pre-norm normalises before each sub-layer. Pre-norm keeps a clean residual path, giving more stable gradients and letting very deep models train without careful warm-up — which is why nearly every modern LLM uses it.

11 · Conditional compute

Mixture-of-experts

A mixture-of-experts (MoE) layer replaces one big feed-forward network with many “expert” FFNs and a router that sends each token to just a few of them. Total parameters grow (more capacity) while the compute per token stays roughly fixed (only the chosen experts run). MoE models get the quality of a much larger dense model at a fraction of the inference FLOPs — at the cost of memory (all experts must be resident) and routing complexity.

Q

MoE — more parameters, same compute

Dense models use every parameter for every token. MoE activates only a small subset (e.g. 2 of 8 experts) per token, so a 50B-parameter MoE can cost like a ~10B dense model to run, while holding far more knowledge. The catch: you still need memory for all the experts, and the router can load-imbalance.

12 · Putting it together

The efficiency stack

No single trick wins; they compose. A typical open-weights deployment quantizes to 4-bit, uses GQA and FlashAttention kernels, a paged KV cache, RoPE positions, and possibly an MoE backbone — served by a runtime like vLLM with continuous batching. Each technique attacks a different bottleneck: memory, bandwidth, or FLOPs.

13 · Self-check

Questions before you move on

What does quantization trade, and where does it break down?

The KV cache speeds up generation by:

Grouped-query attention (GQA) primarily reduces:

A mixture-of-experts layer lets a model have many more parameters while keeping per-token compute roughly fixed because:

14 · Recap

One-screen summary

Chapter 09 — load-bearing ideas

Inference dominates lifetime cost — these techniques make a fixed model cheaper/faster, not more accurate.
Open vs closed weights trade control and privacy against zero-infra convenience; runtimes (vLLM, llama.cpp) hugely affect throughput.
Quantization cuts memory with bit-width; accuracy holds to ~4 bits, then cliffs.
The KV cache turns generation from O(n²) to O(n) by storing past keys/values — paid for in memory.
GQA/MQA shrink the KV cache by sharing key/value heads; FlashAttention is an exact, memory-aware kernel; sliding-window makes attention linear.
RoPE encodes relative position by rotation; pre-norm stabilises deep training.
Mixture-of-experts adds parameters (capacity) without adding per-token compute, via sparse routing.

15 · Exam · past papers

Past-paper questions

Answered 0 / 6 · 0 correct

Q-EFF1Quantizing model weights from FP16 to INT4 primarily:
Q-EFF2The KV cache changes the per-step generation cost from:
Q-EFF3FlashAttention is faster because it:
Q-EFF4Grouped-query attention (GQA) trades:
Q-EFF5Rotary position embeddings (RoPE) encode position by:
Q-EFF6A mixture-of-experts model increases capacity without increasing per-token compute because: