AI Under the Hood

01Data and measurement

Statistics is the science of collecting, organising, analysing and interpreting data. Descriptive statistics summarise the data you have; inferential statistics use a sample to draw conclusions about a larger population.

Types of data

Scales of measurement

The scale decides which statistics, tests and encodings are valid.

• Interval vs ratio: 20 °C isn't “twice as hot” as 10 °C, because 0 °C is arbitrary; 20 kg is twice 10 kg.
• Likert scales (totally agree … totally disagree) are ordinal. They are often averaged as if interval, which assumes equal gaps between answers.

02Descriptive statistics

Central tendency

Dispersion

z-scores and skewness

Worked example

Heights of seven cricketers from one camp: 175, 180, 140, 140, 135, 160, 135 cm.

03Probability and distributions

Probability basics

The sample space Ω is the set of all outcomes (rolling a die: {1, …, 6}); an event is a subset of it. Probabilities lie in [0, 1] and sum to 1 over Ω.

Random variables and distributions

Common distributions

The normal distribution and the empirical rule

Symmetric, bell-shaped and unimodal, with mean = median = mode. Blood pressure, heights and measurement errors are roughly normal, and averages of many values tend towards it (the central limit theorem).

Checking whether data is normal

• Plots: a histogram, and a Q-Q plot of sorted data against normal quantiles: points on a straight line mean normal.
• Tests: Shapiro–Wilk, D'Agostino–Pearson, Kolmogorov–Smirnov. With large n they flag tiny, harmless departures, so look at the plot too.
• If skewed: try a log or Box-Cox transform, or use tests that don't assume normality.

Right-skewed and log-normal features are good candidates for a log transform before training (see feature engineering).

04Sampling and correlation

Sampling techniques

Sampling bias is why a model trained on convenient data can fail on real users: the sample doesn't represent the population (see data curation).

Covariance and correlation

05Inferential statistics

Estimates and the central limit theorem

A point estimate is a single value for an unknown parameter (x̄ for μ); an interval estimate gives a range of plausible values. Intervals rely on the central limit theorem (CLT): whatever the population's shape, the mean of a large enough sample is approximately normal.

Confidence intervals

• Model metrics need intervals too: report “accuracy 0.84 (95% CI 0.82–0.86)”. For metrics without a formula, use the bootstrap: resample the test set, recompute the metric, and take the 2.5th and 97.5th percentiles.

Hypothesis testing

1. State the null hypothesis H₀ (no effect: “the new model is no better”) and the alternative H₁.
2. Choose a significance level α, usually 0.05.
3. Compute a test statistic from the sample.
4. Find the p-value: the probability of a result at least this extreme if H₀ were true.
5. If p < α, reject H₀; otherwise you fail to reject it (which is not proof that H₀ is true).

Choosing a test

Statistics for ML experiments

• Multiple comparisons: test 20 prompts at α = 0.05 and one will look “significant” by chance. Correct with Bonferroni (α / m) or control the false-discovery rate (Benjamini–Hochberg).
• Practical vs statistical significance: decide in advance what improvement matters (the tolerance in a quality gate), then check the CI clears it.
• A/B tests in production (testing in production) are hypothesis tests: fix the sample size from a power calculation before starting, and don't stop early when the p-value first dips below 0.05.
• Comparing two models on the same test set is a paired comparison: use McNemar's test or a paired bootstrap, not two independent intervals.

06Exploratory data analysis (EDA)

Outliers

• Skew the mean (the median and mode are robust) and the standard deviation.
• Affect normalisation, especially min-max scaling, where one extreme value squashes all the others.
• A single outlier can dominate a squared loss such as MSE or RMSE, which hurts generalisation.
• Detect: IQR rule (outside [Q1 − 1.5·IQR, Q3 + 1.5·IQR]) or |z-score| > 3.
• Check before removing: an outlier may be real and important (fraud, failures). Alternatives are clipping, a log transform or a robust loss (Huber).

High cardinality

Categorical features with many values (user IDs, zip codes):

• One-hot encoding explodes the number of dimensions, making training expensive.
• Poor generalisation: the model can memorise individual IDs.
• Fixes: group rare categories into “other”, target or hash encoding, learned embeddings.

07Data curation and quality

Better data often helps more than a better model. Curation decides what goes into the training set and how trustworthy its labels are.

The data pipeline

• Collection: human annotation, automatic labelling, synthetic data.
• Cleaning: remove noise, duplicates and broken examples.
• Transformation and preprocessing: categorical encoding, bins, tokenisation, normalisation (see feature engineering).
• Quality vs quantity: for fine-tuning LLMs, quality usually wins. The LIMA paper fine-tuned on about 1,000 carefully chosen examples with strong results.
• Privacy, bias, licensing and consent.
• Contamination: test or benchmark data leaking into training inflates evaluation scores. Deduplicate against your test sets.

Questions to ask your data

Heuristic preprocessing

• Pandas for analysis and fixes, from outlier removal to imputation.
• Text filters: language ID, length limits, repetition, toxicity, and perplexity filters (text a reference model finds unusually surprising).
• Deduplication: exact (hashing), near-duplicate (MinHash), semantic (embedding similarity).
• Embeddings for label consistency: very similar examples with different labels are likely mislabelled.

Finding label errors with a human in the loop

Cleanlab is a data-centric AI package that flags mislabelled, outlying or ambiguous examples. It uses confident learning: out-of-sample predicted probabilities (from cross-validation) plus a confidence threshold per class estimate which given labels are probably wrong. The predictions must be out-of-sample: a model that trained on an example has memorised its label and can't flag it.

Argilla (now part of Hugging Face) is a collaborative annotation tool. You can send model or generative outputs to its interface for review, or pre-fill suggested corrections above a probability threshold. Apply the same quality rules to your test set, and never let training-time fixes peek at it.

Active learning

Labelling is expensive, so at each round the model tells you which examples to label next, usually the ones it is least sure about.

ActiveLab (from Cleanlab) is useful when human labels are noisy. It weighs annotator agreement against model confidence to decide whether to re-label a suspect example or label a new one.

Synthetic data generation

A policy model generates several candidate outputs (proofs, code, text) per input; a reward model or verifier scores them on correctness or usefulness; the best are kept as training data, and the policy is fine-tuned on them. Repeat until performance plateaus. This is rejection sampling / best-of-n; variants include STaR, ReST and expert iteration.

• ✗ Reward hacking: the policy learns to exploit flaws in the reward model; its scores rise while real quality doesn't.
• ✗ Lower diversity and model collapse: repeatedly training on your own outputs narrows them (see data-centric fixes).
• ✓ Prefer verifiable checks where possible (unit tests for code, proof checkers, exact answers): more reliable than a learned reward model.
• Other methods: self-instruct (the model writes its own instructions and answers), distillation from a stronger model, paraphrasing and back-translation.

08Missing data, imbalance and resampling

Missing data

The right fix depends on why values are missing:

• Simple imputation: mean or mode; the median for skewed data, since it is robust to outliers.
• ML-based imputation: k-NN, iterative imputation (MICE), decision trees. Some models, such as XGBoost, handle missing values natively.
• Add a “was missing” indicator feature: missingness itself can be predictive.
• Fit the imputer on the training set only, the same rule as for scalers.

Imbalanced data

• Oversample the minority class: random copies or SMOTE.
• Undersample the majority class: fast, but throws data away.
• Hybrid: both together.
• Class weights or a weighted loss: give minority errors a larger penalty. Focal loss focuses training on hard examples.
• Threshold tuning: move the decision threshold away from 0.5.
• Evaluate with the right metric: F1, PR-AUC or recall, not accuracy.

Resampling methods

• Stratified splits: preserve class proportions in every split.
• k-fold cross-validation: use the stratified version for imbalanced data.
• Bootstrapping: draw n points with replacement from n. Used for confidence intervals on a metric and for bagging (random forests).

09Feature engineering

Scaling numerical features

If feature X spans 154 to 24,917,482 and feature Y spans 5 to 22, X's gradients swamp Y's (see section 13). Normalisation helps models converge faster, make better predictions, avoid the “NaN trap” from overflowing values, and learn a sensible weight for each feature.

• Fit the scaler on the training set only, then apply it to validation and test data. Fitting on all data leaks test information.
• Tree-based models (random forests, XGBoost) don't need scaling: splits ignore scale.

Binning (bucketing)

Convert a numerical value into ranges. Useful when the relation between feature and label is non-linear or weak overall, or when values form clusters. Quantile binning puts the same number of points in each bin.

Categorical features

One-hot encoding for few categories, learned embeddings for many, and feature crosses (combinations such as city × hour) to capture interactions.

10Feature importance and dimensionality

• Random-forest importance ranks features by how much they reduce impurity, but is biased towards continuous and high-cardinality features. Permutation importance (shuffle a feature, measure the drop in score) and SHAP values are more reliable.
• Correlation analysis finds redundant features: if two are highly correlated, one may be enough.
• PCA reduces dimensionality by building new features (principal components) from the eigenvectors of the covariance matrix. Especially useful with correlated features. It is not a feature-importance method, and components lose interpretability. Standardise first, or large-scale features dominate.

11From likelihood to loss to gradients

MLE defines the objective → the loss function encodes it → gradients tell us how to reach it.

Maximum likelihood estimation (MLE)

Given data points (x⁽ⁱ⁾, y⁽ⁱ⁾) drawn from a conditional distribution P(y | x; θ), choose the parameters θ (e.g. w, b) that make the observed data most probable. Products are inconvenient, so take the log:

12Evaluation metrics

Regression

• R² can be negative on test data, when the model is worse than always predicting the mean.
• R² never decreases when you add features; use adjusted R² to compare models with different numbers of features.

Classification

Clustering

Clustering metrics need no labels. Inertia measures compactness; silhouette measures separation.

13How outliers and unscaled features affect training

Effect on gradients

Outliers can distort the updates. With MSE regression, the error (ŷ − y) becomes very large when the model is badly wrong, pulling the weights towards fitting the outlier. In logistic regression (ŷ − y) is bounded between −1 and 1, so large feature values cause the big gradients, and mislabelled points keep pulling with maximum force.

• Mitigations: detect and handle outliers before training; robust losses (Huber); gradient clipping. Regularisation limits weight size but doesn't remove an outlier's pull, so it is a weaker fix.

Unscaled features cause unstable, slow training. The update for each weight is proportional to its feature's values, ∂J/∂wⱼ = (1/m) Σ (ŷ⁽ⁱ⁾ − y⁽ⁱ⁾) xⱼ⁽ⁱ⁾, so a feature in the thousands dominates one in [0, 1]. The loss surface becomes a long, narrow valley.

• Mitigation: feature scaling (z-score to mean 0, std 1; or min-max to [0, 1]).

Effect on the decision boundary

• Outliers: dangerous even when data is separable. A single extreme point can tilt the boundary, worst for hard-margin SVMs. The classifier still separates the data, but generalisation suffers.
• Separable data and logistic regression: without regularisation the weights grow without limit, because the loss keeps decreasing as the boundary gets sharper; the MLE doesn't exist. Regularisation or early stopping keeps the weights finite.
• Normalising features: linear rescaling doesn't change whether data is separable. For an unregularised model trained to convergence the boundary is equivalent, just with different weights. With L2 regularisation, early stopping, SVM margins or distance-based models (k-NN, k-means), scaling changes the boundary itself. The result of normalising is faster, more stable training and, with regularisation, often a better boundary.

14The i.i.d. assumption and dependent data

Averaging the loss over a batch treats every sample equally. That assumes samples are independent and identically distributed (i.i.d.) draws from the true data distribution.

With dependent data (time series, sequences, graphs) the problems are:

• Correlated mini-batches: consecutive samples carry nearly the same information, so the gradient estimate has high variance and can be biased towards the current stretch of the sequence.
• Leakage when splitting: a random train/test split lets the model see the “future”. Split by time instead (forward chaining, see cross-validation).

How to handle it:

• Weight samples or the loss.
• Model the dependencies explicitly: RNNs, transformers, conditional random fields.
• Change the sampling: draw windows from different parts of the sequence, with gaps between them, in each mini-batch.

15Bias and variance

Bias is error from wrong or overly simple assumptions. A high-bias model underfits: it misses real patterns and does badly on training and test data alike.

Variance is error from being too sensitive to the particular training sample. A high-variance model overfits: it learns the noise, does well on training data and badly on new data, and would change a lot if trained on a different sample.

For squared error, the expected test error splits into three parts:

How to diagnose it

16Overfitting

The model learns the training data too well, including its noise and outliers (memorisation). Training error is very low, validation error is high.

Causes

1. Model too complex for the amount of data.
2. Too little training data.
3. Training for too many epochs.
4. Noisy labels, which give the model noise to memorise.

Remedies fall into two groups: change the data, or change the model and training.

17Data-centric fixes

1. Collect more data. More examples make memorising noise harder and shrink the train–validation gap. Helps against overfitting, not underfitting.
2. Data augmentation or synthetic data. Augmentations must keep the label valid: flipping a cat photo is fine, flipping a “6” into a “9” is not. Low-quality synthetic data can add bias, and training LLMs heavily on model-generated text can cause model collapse (not the same as GAN mode collapse).
3. Self-supervised or transfer learning. The model learns general features from large unlabelled or related data, so it needs fewer labelled examples. Transfer learning is partly model-centric too.
4. Clean labels, remove noise and outliers. Less noise to memorise (tools: section 7).
5. Deduplicate. Especially important for LLMs: reduces memorisation and prevents train/test leakage.
6. Feature selection. Fewer irrelevant features, fewer spurious patterns.

18Model-centric fixes

A. L2 and L1 regularisation

Add a penalty on weight size to the loss: total loss = standard loss (prediction error) + regularisation term.

• λ sets the trade-off. Too high: weights are forced near zero and the model underfits (training still converges, just to a too-simple solution). Too low: little effect.
• Bias terms are usually not penalised.
• With plain SGD, L2 and weight decay are the same thing. With Adam they are not; see AdamW.

B. Dropout

During training, each activation is set to zero with probability p, so neurons can't rely on specific partners. At inference all neurons are used (activations are scaled to compensate).

C. Early stopping

• Monitor validation loss.
• Use a patience window (stop after N epochs without improvement) so random noise doesn't trigger a stop.
• Restore the best checkpoint, not the last one.

D. Fewer trainable parameters

• Pruning: train the full model, remove weights (unstructured, usually by smallest magnitude) or whole neurons, heads or channels (structured), retrain, and repeat.
• Knowledge distillation: a small student learns to match a large teacher's softened output distribution (full treatment in section 51).
• LoRA for fine-tuning: train a small low-rank update instead of all weights. See section 52.

E. Ensembles

Combine several models. More common in classical ML than with large neural networks, because of cost.

F. Normalisation layers

These mainly stabilise and speed up training. Any regularising effect is indirect and weaker than dropout or weight decay.

Where the norm sits also matters: the original Transformer normalised after each residual add (post-LN); modern LLMs normalise before each sub-layer (pre-LN), which trains more stably. See section 37.

19Cross-validation

k-fold cross-validation is an evaluation and model-selection method, not an ensemble and not a remedy by itself. It detects overfitting and helps tune hyperparameters such as λ, dropout rate or number of epochs.

• k-fold: each fold holds out 1/k of the data, not k samples.
• Leave-one-out (LOOCV): the special case k = n.
• Leave-p-out: holds out p samples in each round.
• Stratified k-fold keeps class proportions in each fold; for time series use forward-chaining splits.
• Bootstrapping (sampling with replacement) is the other common resampling method: see section 8.

20Beyond the classic picture

Double descent

Test error first follows the classical U-curve, then peaks at the interpolation threshold: the point where the model has just enough capacity to fit the training data exactly (roughly, parameters ≈ data points). Past that point, error falls again as the model grows and can end below the best classical model.

• The peak is worse with noisy labels; regularisation and early stopping reduce it.
• It also appears over training time (epoch-wise) and with dataset size (sample-wise).
• It helps explain why huge LLMs generalise despite having billions of parameters.

Grokking

The model first memorises the training set (≈100% training accuracy, chance-level validation accuracy). After many more steps, validation accuracy suddenly jumps to near-perfect. The smaller the training set, the more optimisation steps this takes.

• Weight decay is a key ingredient.
• Practical lesson: early stopping can end training before the model groks.

21Underfitting

The model is too simple, or trained too little, to capture the underlying patterns (high bias). Training and validation error are both high and close to each other.

Causes

• Model too simple for the task.
• Too few training epochs.
• Too much regularisation or dropout.
• Poor or missing features.
• Bad optimisation, e.g. a learning rate that is too low or too high.

Fixes

• Increase model complexity: more layers or units, or a more expressive model class.
• Train longer.
• Feature engineering: more relevant features, interaction terms.
• Reduce regularisation: lower λ, lower dropout rate p.
• Boosting, which reduces bias. Bagging reduces variance and does little here.
• Tune optimisation: learning rate, a better optimiser such as Adam, normalisation layers (Part II).
• Clean the data: noisy labels limit how well any model can do (a weaker, indirect fix).

22Loss landscapes: convex and non-convex

A function is convex if the straight line (chord) between any two points on its graph never goes below the graph:

• Key property: in a convex function every local minimum is a global minimum. That is what makes convex optimisation easy.
• A convex function can have many global minima (a flat-bottomed valley) or none (eˣ, a straight line). Strictly convex functions (f'' > 0) have at most one.
• Non-convex simply means not convex. It may have several local minima, but not necessarily.

Saddle points

A saddle point has zero gradient but curves up in one direction and down in another, like f(x, y) = x² − y². It only really exists in two or more dimensions; a flat inflection point is the closest 1D analogue.

What this means for deep learning

• Convex losses: MSE for linear regression, cross-entropy for logistic regression, hinge loss for SVMs.
• Neural-network losses are non-convex because stacked non-linear layers make them so, even with MSE or cross-entropy. Convexity depends on the model plus the loss, not the loss formula alone.
• In high dimensions, saddle points and plateaus are far more common than bad local minima, and most local minima have similar loss. The real difficulty is getting through saddles and flat regions; SGD noise, momentum and Adam all help.

Weight initialisation

Good initialisation doesn't guarantee the best minimum. It matters for two other reasons:

• Breaking symmetry. If all weights start at zero, every neuron in a layer gets the same gradient and learns the same thing.
• Keeping signals at a healthy scale through the layers, so gradients don't vanish or explode. Xavier/Glorot initialisation suits tanh and sigmoid; He initialisation suits ReLU.

23Gradient descent

Goal: find the parameters that minimise the loss, w* = argmin_w L(w). Start from initial values and repeatedly step against the gradient.

Batch, stochastic and mini-batch

Mini-batch is the standard choice. In deep learning, “SGD” almost always means mini-batch SGD.

24Momentum

Add a velocity that accumulates past gradients, like a ball rolling downhill. Steps build up along directions where the gradient is consistent and cancel where it zig-zags.

25Adaptive optimisers: Adam and AdamW

Adam (Adaptive Moment Estimation)

• ✓ Works well across many problems.
• ✓ Little tuning of β₁, β₂, ε. The learning rate still needs tuning.
• ✗ Needs 2 extra values per parameter (m and v): optimiser memory ≈ 2× model size.
• ✗ On some tasks (e.g. image classification) well-tuned SGD + momentum generalises slightly better; can be unstable early in training (one reason for warmup).

Why Adam + L2 goes wrong, and AdamW fixes it

26Learning-rate schedules

Warmup

Start with a small learning rate (small updates, not small weights) and raise it over the first few hundred to few thousand steps.

• Early on, weights are random, so gradients are large and unreliable. A full learning rate would cause destabilising jumps.
• Adam's estimates m and v are poor in the first steps; warmup gives them time to settle.

Cosine decay

27Gradient clipping

Prevents single huge steps caused by exploding gradients, which lead to instability and divergence. Common in RNNs and transformers.

28Generative vs discriminative

The families below differ in how they represent p(x): through a latent variable (VAE), a two-player game (GAN), an invertible mapping (flow), a chain of conditionals (autoregressive), or iterative denoising (diffusion).

29Variational autoencoders (VAEs)

A normal autoencoder maps each input to a single point in a lower-dimensional latent space. A VAE maps each input to a probability distribution over the latent space, usually a Gaussian with a mean μ and standard deviation σ per latent dimension. The decoder takes a sample z from that distribution and reconstructs the input.

Loss: the negative ELBO

Reparameterisation trick

• Training: z is sampled from the encoder's q_φ(z|x).
• Generation: z is sampled from the prior N(0, I) and decoded.
• ✓ Stable training, good coverage of the data, a useful latent space. Stable Diffusion uses a VAE to compress images before diffusion.
• ✗ Samples tend to be blurrier than GAN or diffusion samples.

30Generative adversarial networks (GANs)

Two networks compete. The generator turns noise into fake samples; the discriminator tries to tell real from fake. As the generator improves, the discriminator must get sharper, and vice versa, until the discriminator can no longer tell them apart.

• ✓ Sharp, high-quality samples.
• ✓ Fast generation: one forward pass.
• ✗ Unstable training and sensitive hyperparameters; resource-hungry.
• ✗ Mode collapse: the generator produces only a few kinds of outputs and ignores the rest of the data distribution (low diversity).
• ✗ No explicit likelihood: you cannot compute p(x).

31Flow-based models

A chain of invertible transformations turns a simple distribution (Gaussian noise) into the complex data distribution. Because every step can be inverted, the model gives an exact log-likelihood.

• ✗ Design constraints: every layer must be invertible, keep the same dimensionality, and have a cheap Jacobian determinant. Sample quality has historically lagged GANs.
• Examples: RealNVP, Glow. A related modern idea, flow matching, is used in recent image models such as Stable Diffusion 3.

32Autoregressive models

Factorise the distribution into a chain of conditionals and generate one element at a time, each conditioned on everything generated so far.

33Diffusion and consistency models

Diffusion models

A forward process adds Gaussian noise over T steps until only noise is left. It is fixed; nothing is learned. A network learns the reverse process: removing noise step by step, which turns fresh noise into new data.

• ✓ Very high quality and diversity; stable training with no mode collapse.
• ✗ Slow sampling: tens to hundreds of network passes. Fixes: DDIM samplers, distillation, and latent diffusion, which runs diffusion in a VAE's compressed latent space (Stable Diffusion).

Consistency models

Introduced by Song et al. (2023). A consistency model learns a function that maps any point on a noise-to-data trajectory to the same clean sample, so it can jump from noise to data in one step.

• Consistency distillation: trained from an existing diffusion model. Consistency training: trained from scratch.
• Generate in one or a few steps. Quality is slightly below many-step diffusion; adding steps trades speed for quality.

34Comparing generative models

These ratings are rules of thumb; they shift with scale and recent research.

35Embeddings and positions

Distributional hypothesis: words that occur in the same contexts tend to have similar meanings (“you shall know a word by the company it keeps”, Firth). Embeddings put this into practice: each token becomes a vector, and tokens used in similar contexts end up close together.

• Similarity is usually measured with cosine similarity: cos(a, b) = a·b / (‖a‖ ‖b‖).
• word2vec and GloVe give one static vector per word. Transformers produce contextual embeddings: “bank” gets different vectors in “river bank” and “bank account”.

Adding position information

Attention on its own treats the input as a set, so word order would be lost. Position information is added in one of three ways:

36Attention

Self-attention

Each token builds a contextual representation by weighing every other token in the same sequence, all positions in parallel. In self-attention, queries, keys and values all come from the same sequence X:

Causal (masked) attention

The prediction at each position may depend only on the current and previous positions, never on future ones. Scores for future positions are set to −∞ before the softmax, so their weight becomes 0 and each row still sums to 1.

Multi-head attention

Run several attentions in parallel, each with its own learned projections, then concatenate the results and mix them with an output projection. Each head can focus on a different kind of relation (syntax, coreference, nearby words) in a different representation subspace.

37The Transformer architecture

The original Transformer (Vaswani et al., 2017) is an encoder–decoder, 6 layers each. The encoder reads the input and outputs one contextual vector per token. The decoder generates the output one token at a time, using its own previous tokens and the encoder's output.

• Encoder layer: multi-head self-attention, then a feed-forward network.
• Decoder layer: causal self-attention; cross-attention (queries from the decoder, keys and values from the encoder); a feed-forward network.
• Output: a linear layer and a softmax over the vocabulary give next-token probabilities.
• Positions are added at the input (see section 35).
• Post-LN vs pre-LN: the original normalises after each residual add (post-LN). Modern LLMs normalise before each sub-layer (pre-LN), usually with RMSNorm, which trains more stably.

38Model families

Encoder-only models see the whole input at once, so they don't generate text autoregressively. They produce task outputs such as labels or per-token predictions.

39Mixture of experts (MoE)

MoE replaces the single feed-forward network (FFN) in a transformer block with many expert FFNs plus a router. Each token is sent to only a few experts, so the model gets far more parameters (more capacity to store knowledge) without a matching increase in compute per token. This is called sparse or conditional computation.

How a token is routed

1. Router input: the token's hidden state x. The router is usually a linear layer followed by a softmax, trained together with the rest of the network. It decides before seeing any expert output.
2. Expert computation: the selected experts each process the same x with their own weights.
3. Combination: their outputs are summed, weighted by the router's scores, and passed on to the next layer.

• MoE layers may replace every FFN or only some. DeepSeek-V3, for example, keeps its first 3 layers dense.
• Experts tend to specialise in token types and syntax patterns rather than clear topics such as “maths” or “biology”.

Total vs active parameters

Expert designs

• (a) Conventional top-2: a few large experts, 2 active per token.
• (b) Fine-grained experts: split each expert into smaller ones and activate more of them. Compute stays the same, but the number of possible expert combinations explodes: choosing 2 of 16 gives 120 combinations, choosing 8 of 64 gives about 4.4 billion. This allows more specialised experts.
• (c) Shared expert: one or more experts are always active and learn common knowledge, so the routed experts don't each have to relearn it. This reduces redundancy.

Load balancing

Left alone, the router sends most tokens to a few popular experts. Those experts train faster, so they get picked even more: a self-reinforcing loop that wastes the other experts (expert collapse). Two separate mechanisms counter this:

• Router z-loss (ST-MoE) penalises large router logits to keep training stable.
• Auxiliary-loss-free balancing (DeepSeek-V3) adds a per-expert bias to the routing scores, raised for under-used experts and lowered for over-used ones, avoiding the quality cost of an extra loss term.

Overfitting and fine-tuning

Sparse models have huge capacity relative to small fine-tuning datasets, so they are prone to overfitting (high variance), and historically generalised poorly after fine-tuning.

• Use a higher dropout rate inside the experts than in the dense layers.
• Fine-tune fewer parameters, e.g. LoRA or only the non-expert layers.
• Instruction tuning: MoE models were found to benefit from it more than dense models (Shen et al., 2023).

Dense or MoE?

40Efficient attention and the KV cache

The KV cache

During generation, the keys and values of earlier tokens don't change, so they are stored instead of recomputed. The cache grows with every token and quickly becomes the main memory cost of long contexts:

MHA, GQA and MQA

Grouped-query attention lets groups of query heads share one key–value head. Query heads stay separate; only K and V are shared.

• The gain is mainly at inference: a smaller cache and less memory traffic, which matters for long contexts and large batches. It saves little training compute.
• GQA is used in Llama 2 70B, Llama 3 and Mistral.

Multi-head latent attention (MLA)

Introduced in DeepSeek-V2 and used in V3/R1. Instead of sharing K and V, MLA compresses them into a small latent vector and caches only that.

1. Down-project each token to a latent vector: c_KV = W_DKV · x, with d_c ≪ n_heads × d_head.
2. Cache only c_KV.
3. Up-project when needed: K = W_UK · c_KV, V = W_UV · c_KV.

• The up-projection can be skipped at inference: W_UK can be merged into the query projection and W_UV into the output projection, so full K and V are never built.
• RoPE complication: rotary position embeddings don't survive the compression, so MLA keeps a small separate key part that carries position information, and caches it too.
• GQA vs MLA: GQA shares KV heads at a small quality cost; MLA compresses them, and DeepSeek reports quality on par with or better than MHA with a much smaller cache.

Other efficiency techniques

• FlashAttention: computes exact attention faster by reducing reads and writes to GPU memory. An implementation speed-up, not an approximation.
• Sliding-window attention: each token attends only to the last w tokens, making cost linear in sequence length. Used in Mistral.

41State space models

SSMs are a sequence architecture, an alternative to attention, not a separate type of generative model. They can be the backbone of an autoregressive generative model. They model an internal hidden state that evolves over time and is usually not observed directly.

42The memory budget

Each training step has three phases: forward pass, backward pass, optimizer step. GPU memory has to hold four things:

• Parameters: the learnable weights.
• Gradients: one per parameter, used to update it.
• Optimizer states: for Adam, the momentum m and variance v of every parameter.
• Activations: layer outputs created in the forward pass and kept for the backward pass.

What drives memory and compute

• Batch size: activation memory grows linearly with it. Larger batches give more accurate gradients, better hardware use and fewer optimizer steps per epoch. Very large batches can hurt generalisation unless the learning rate is scaled.
• Sequence length: activations grow quadratically with naive attention (the n×n score matrix) and linearly with FlashAttention.
• Precision: 32-bit vs 16-, 8- or 4-bit values. Lower precision saves memory but can cost accuracy.
• Model size: number of layers and their width.

43Saving memory on one GPU

Gradient checkpointing (activation recomputation)

Instead of storing every activation for the backward pass, keep only a few checkpoints. When the backward pass needs a missing activation, recompute it from the nearest checkpoint.

• Selective recomputation recomputes only the parts that are cheap to compute but expensive to store.
• FlashAttention recomputes only the attention scores in the backward pass instead of storing the n×n matrix. That is a narrow form of recomputation, not general checkpointing.

Gradient accumulation

Run several small micro-batches forward and backward, add up their gradients, then take one optimizer step. This simulates a large batch without holding it in memory.

Mixed-precision training

Do most computation in 16-bit (BF16 or FP16) and keep FP32 master weights and optimizer states for accuracy. Gradients are usually BF16.

• FP16 needs loss scaling: its small range would round tiny gradients to zero. BF16 has FP32's range, so it doesn't.
• FP8 training is the newest step; DeepSeek-V3 used it.
• In PyTorch and Hugging Face this is autocast / AMP (the bf16=True or fp16=True Trainer flags).

Quantization

Store weights (and sometimes activations) in fewer bits, e.g. 32-bit floats to 8-bit integers, using rounding and sometimes outlier clipping. Mainly used for inference and QLoRA fine-tuning; mixed precision is the training technique.

• Post-training quantization (PTQ) quantizes a finished model: GPTQ and AWQ (weight-only), bitsandbytes LLM.int8 and 4-bit NF4.
• Quantization-aware training (QAT) simulates quantization during training so the model learns to tolerate it.

Attention slicing

Computes attention in chunks one after another on a single GPU, lowering peak memory at the cost of speed. It comes from the diffusers library and is an inference trick, not a multi-GPU technique.

44Parallelism across GPUs

Data parallelism (DP)

Each GPU holds a full copy of the model and processes a different mini-batch. After the backward pass, gradients are averaged with an all-reduce so every copy stays in sync.

• ✓ Nearly linear speed-up when communication is fast (NVLink, NCCL). The all-reduce is overlapped with the backward pass, with gradients grouped into buckets.
• ✗ Per-GPU memory is not reduced: each GPU stores the whole model and optimizer state. Gradient sync becomes a bottleneck on slow networks or very large models. ZeRO fixes the memory problem.

Pipeline parallelism (PP)

Naive model parallelism splits the layers into chunks on different GPUs and passes activations from one to the next. It lets a model larger than one GPU train, but only one GPU works at a time. Pipeline parallelism fixes this by splitting each batch into micro-batches that flow through the stages, so GPUs work on different micro-batches at once.

• GPipe: all forward passes, then all backward passes (as in the figure).
• 1F1B: alternates one forward and one backward, so fewer activations are held at once.
• Interleaved: each GPU holds several smaller stages; a smaller bubble at the cost of more communication.

Tensor parallelism (TP)

Splits the matrices inside each layer across GPUs, so every GPU works on the same tokens with a slice of the weights. Suited to wide layers; deep models are pipeline parallelism's job.

• TP communicates activations on every layer, so keep it inside one node on fast NVLink, typically TP ≤ 8.
• TP shards activations only inside the split layers. LayerNorm and dropout still hold full activations, so sequence parallelism is added alongside TP to shard those along the sequence.

Context and expert parallelism

• Context parallelism (CP): splits a long sequence across GPUs (e.g. ring attention), for very long contexts.
• Expert parallelism (EP): places MoE experts on different GPUs and sends tokens to them.

DP, TP, PP, CP and EP together are often called 5D parallelism.

Communication operations

45ZeRO: sharding the model state

Plain data parallelism stores the same parameters, gradients and optimizer states on every GPU. ZeRO (DeepSpeed) removes this redundancy by partitioning them across the data-parallel GPUs, so each stores only a slice.

Ψ = number of parameters, N = data-parallel GPUs, mixed precision with Adam.

• PyTorch FSDP (Fully Sharded Data Parallel) is essentially ZeRO-3.
• Limits: ZeRO doesn't shard activations, which grow with batch size and sequence length. ZeRO-3 also needs each layer's full parameters gathered on one GPU while that layer runs. Tensor, sequence and context parallelism address these.

46Combining strategies

• Data + pipeline: each data-parallel replica is a pipeline spread over several GPUs.
• Data + tensor: TP inside each node (fast NVLink), DP across nodes. Each data-parallel replica is a TP group.
• 3D parallelism (DP + PP + TP): typically TP inside a node, PP across nodes, DP outermost. DeepSpeed and Megatron combine these.
• ZeRO-1, not ZeRO-3, with pipelines: ZeRO-2/3 would have to communicate gradients or parameters for every micro-batch, which clashes with pipelining.

What each technique reduces

Further reading: Hugging Face, The Ultra-Scale Playbook (2025).

47Decoding strategies

Autoregressive decoding generates one token at a time: the model outputs a probability for every vocabulary token, one is chosen, appended to the input, and the loop repeats until an end token or a length limit. Each step reuses the KV cache (section 40). Strategies are either deterministic (search) or stochastic (sampling).

Deterministic: greedy and beam search

• Greedy search: always take the most likely token. Simple and fast, but it misses better sequences hidden behind a lower-probability first token, and can be repetitive.
• Beam search: at each step, extend every beam with every possible next token, score each sequence by its total probability, and keep the top B (the beam width). It is a heuristic: it doesn't guarantee the most probable sequence overall.

• Good for translation and summarisation; poor for open-ended chat, where it produces generic, repetitive text. An n-gram repetition penalty helps.
• Contrastive search is a deterministic alternative that penalises tokens too similar to earlier context.

Stochastic: top-k, top-p and min-p

• Top-k: keep the k most probable tokens, renormalise them to sum to 1, and sample. Lower k is more predictable, higher k more diverse.
• Top-p (nucleus): sort tokens by probability and keep the smallest set whose cumulative probability reaches or exceeds p, then renormalise and sample. Typical p = 0.9–0.95.
• Min-p: keep tokens whose probability is at least a fraction of the top token's. Adapts like top-p and holds up well at high temperature.

• Order of operations in most libraries: temperature first, then top-k, then top-p, then sample. Settings like temperature=0.7, top_p=0.9 combine.
• Repetition / frequency penalties down-weight tokens that have already appeared.
• Sampling (temperature > 0) is what makes self-consistency and best-of-N possible.

Temperature

Temperature rescales the logits before the softmax. It is an inference setting for sampling; it doesn't affect training optimisation. It also appears in knowledge distillation.

48Test-time compute

Instead of only growing pretraining budgets, test-time methods let a trained model “think longer” on harder problems by spending more compute at inference.

• Reasoning models (OpenAI o1, DeepSeek-R1) are trained with reinforcement learning to produce one long chain of thought: sequential scaling built into the model.
• Compute-optimal scaling (Snell et al., 2024): the best strategy depends on problem difficulty, and a small model with more test-time compute can match a much larger one on some problems. Hugging Face reports Llama 3.2 3B with search outperforming Llama 3.1 70B on MATH-500.

Self-refinement

Self-Refine (Madaan et al., 2023) uses the same model to generate, critique and rewrite, with no extra training: only a feedback prompt, a refine prompt and a stopping rule.

• Stopping rule: a fixed number of rounds, or the model saying no more changes are needed.
• ✗ Works with strong models, but LLMs often can't fix their own reasoning errors without external feedback (Huang et al., 2023) and may change correct answers to wrong ones.
• ✓ Better: get feedback from a separate LLM judge, or even better from a tool or test (a compiler, unit tests, a calculator).

Self-consistency (majority voting)

Sample N answers with temperature > 0 and take the most common final answer. No verifier is needed, but answers must be comparable (numbers, multiple choice). A strong, cheap baseline to beat before adding a verifier.

Search against a verifier

Generate several candidates and let a verifier choose. A verifier can be anything from a hard-coded check to a learned reward model:

• Best-of-N: generate N full solutions and keep the highest-scored one. Weighted best-of-N sums the scores of solutions reaching the same answer and usually wins.
• Beam search: keep N beams; at each step the PRM keeps the top N/M partial solutions and each is expanded into M continuations.
• DVTS (diverse verifier tree search): split the N beams into N/M independent subtrees, each searched greedily with the PRM, which keeps solutions diverse.
• When each wins (Hugging Face results): beam search at small budgets and hard problems; DVTS at large N on easy and medium problems, where beam search collapses onto similar solutions.
• ✗ Cost grows linearly with N, and at large N search finds solutions the verifier likes but that are wrong (reward hacking).

Reducing step scores to one score

A PRM gives one score per step, so a solution needs a reduction to a single number:

• min is pessimistic: one weak step sinks the whole solution.
• prod penalises long solutions: more steps means more factors below 1.
• last worked best in Hugging Face's experiments with their PRM.

Further reading: Hugging Face, Scaling test-time compute with open models (2024).

49Serving LLMs with vLLM

Test-time methods multiply the number of generated tokens, so serving speed matters. vLLM is a fast, easy-to-use library for LLM inference and serving (not to be confused with a VLM).

PagedAttention

Stores the KV cache in fixed-size blocks, like pages of virtual memory in an operating system, instead of one contiguous slab per request.

• Almost no memory is lost to fragmentation, so more requests fit at once.
• Samples from the same prompt share its KV blocks: ideal for best-of-N and self-consistency.
• Prefix caching reuses blocks for shared system prompts across requests.

Continuous batching

Speculative decoding

A small draft model proposes k tokens; the large target model checks all of them in one forward pass. Accepted tokens are kept and the first rejected one is replaced, so generation is faster while the output distribution is unchanged.

Speculative decoding in depth

Goal: produce tokens exactly as if sampled from the target q, while most of the work is done by the cheap draft p. For each draft token x:

while len(seq) < max_len:
    # draft: propose k tokens, keeping each FULL distribution p_1..p_k
    # target: ONE pass on seq + drafts → q_1..q_{k+1}
    #         (logits at position t predict token t+1: shift by one)
    for i in 1..k:
        if rand() < min(1, q_i(x_i) / p_i(x_i)):
            accept x_i
        else:
            append sample(normalise(max(0, q_i − p_i)))
            break
    else:
        append sample(q_{k+1})        # bonus token: all k accepted

50Latency optimisation

Many speed-ups cost no quality at all. Know which phase is slow, then pick the technique that targets it.

Prefill and decode

• Latency is how long one request takes; throughput is tokens per second across all users.
• Decoding at small batch sizes is memory-bound, so anything that moves fewer bytes per step (quantization) or produces more tokens per pass (speculative decoding) speeds it up.

Lossless vs lossy

Model-side techniques

• KV cache: stores past keys and values so they aren't recomputed each step (section 40). PagedAttention is a way to manage that cache's memory, not the cache itself.
• Static cache + torch.compile: torch.compile fuses PyTorch code into optimised kernels but needs fixed tensor shapes; a cache that grows each step would trigger recompilation. Pre-allocating the cache to the maximum length keeps shapes fixed, so compiled kernels and CUDA graphs are reused.
• Speculative decoding (section 49): a small assistant model drafts tokens that the large model verifies.
  • Self-speculative decoding (e.g. LayerSkip): the same model exiting at an early layer drafts; the full model checks. The draft is lower quality, the final output isn't, and no second model is needed.
  • Multi-token prediction: extra heads trained to predict several future tokens (Medusa heads, DeepSeek-V3's MTP module) can serve as the draft.
• Quantization: store weights in lower precision. Weight-only methods (GPTQ, AWQ) speed up decoding mainly because fewer bytes are read per token. Mixed-precision quantization keeps sensitive parts higher-precision (LLM.int8 keeps outlier features in FP16) for better accuracy.
• Smaller models: pruning and distillation. Lossy, so check quality on your task.

Prompt lookup decoding

Speculative decoding where the draft comes from n-grams already in the prompt instead of a draft model: free, and lossless because the model still checks every token. Works best when the output copies from the input: summarisation, RAG answers, code edits.

GPU-side techniques

FlashAttention splits attention into tiles that fit in fast on-chip memory (SRAM) and never writes the n × n score matrix to slower GPU memory (HBM). It is IO-aware and exact.

• FlashAttention-2 divides the work better and also parallelises over sequence length; FlashAttention-3 targets H100 (Hopper) GPUs.
• PyTorch SDPA (scaled_dot_product_attention) automatically picks a backend (FlashAttention, memory-efficient attention, or the plain maths version) depending on GPU, data type and mask.
• Tensor parallelism splits each layer across GPUs, so one request runs faster: it lowers latency for large models but needs fast interconnects (NVLink).
• Batching runs many requests together: it raises throughput but can make each request slightly slower. Continuous batching reduces that cost.
• Vectorisation and kernel fusion: operate on whole tensors instead of loops, and fuse several operations into one GPU kernel to avoid writing intermediates to memory and cut launch overhead. torch.compile and FlashAttention both do this.

Serving-level techniques

• Prefix caching: reuse the KV cache for shared system prompts.
• Chunked prefill: split long prompts so they don't stall other users' decoding.
• Streaming: doesn't make generation faster, but users see text after TTFT instead of waiting for the whole response.

Which technique helps which metric

Further reading: Hugging Face Transformers documentation, LLM inference optimization.

51Knowledge distillation

A small student learns from a large teacher, giving most of the teacher's quality at a fraction of the cost. Three things can be transferred: outputs, intermediate features, or relations.

Response-based: soft targets

Train the student to match the teacher's output distribution (Hinton et al., 2015):

Feature-based

Match intermediate representations (FitNets, Romero et al., 2015), usually with MSE or cosine loss on chosen “hint” layers.

• Student layers are usually narrower: add a small learned projection to map them to the teacher's width.
• Matching very deep layers over-constrains a small student; middle layers are a common choice.
• In practice the losses are combined: L = α·CE + β·KD + γ·L_feature.
• MSE, MAE and RMSE are training losses here. To evaluate a student's embeddings, use cosine similarity to the teacher's, downstream retrieval or classification accuracy, or benchmarks such as MTEB.

Relation-based

Transfer how samples or layers relate to each other, rather than the values themselves.

• RKD (Park et al., 2019) matches distances and angles between samples' embeddings; similarity-preserving KD matches the batch's n × n similarity matrix; FSP matrices capture correlations between two layers' feature maps.
• ✓ Relation matrices have the same shape whatever the model width, so no projection is needed, even across architectures.
• ✗ Choosing which relations to transfer is task-specific, and batch relations cost O(n²) in batch size. Students can approach the teacher; beating it is uncommon.

Training approaches

• Offline: the teacher is trained first and frozen; its soft labels can be precomputed and cached. The standard setup.
• Online: teacher and student train together. In deep mutual learning, peer models learn from each other with no pre-trained teacher.
• Self-distillation: one network is both teacher and student. Deeper layers can teach shallower ones (enabling early exit), or a model can learn from its own earlier predictions. Born-Again Networks train a same-size student that often beats its teacher.

Distilling LLMs

• DistilBERT is 40% smaller and 60% faster than BERT while keeping 97% of its language-understanding performance.
• On-policy distillation (GKD): the teacher scores text the student generates, fixing the train/inference mismatch (exposure bias).
• Licences: some model terms forbid using their outputs to train competing models.

Forward vs reverse KL

• Forward KL(teacher ‖ student) is mean-seeking: the student spreads probability over everything the teacher might say, including the low-probability gap between modes.
• Reverse KL(student ‖ teacher) is mode-seeking: the student commits to the teacher's main answers. Often better for generation (MiniLLM).

52LoRA (Low-Rank Adaptation)

Instead of fine-tuning a full weight matrix W ∈ ℝ^(d×k), freeze it and train a small low-rank update.

• r (rank): higher r gives more expressive updates but more parameters and memory. Typical values: 4 to 64.
• α (lora_alpha): scales the update. Larger α gives stronger updates. Dividing by r lets you change the rank without retuning the learning rate; common choices are α = r or α = 2r.
• Initialisation: A is random (Gaussian), B is zero, so ΔW = 0 at the start and training begins exactly from the pretrained model.
• Parameter savings: r·(d + k) instead of d·k. With d = k = 4096 and r = 8: 65,536 versus 16.8M, about 0.4%. Memory is saved mainly on gradients and optimiser states (Adam's m and v).
• No extra inference cost: after training, merge W′ = W + (α/r)·BA, or keep adapters separate and swap them per task.
• Where: the original paper (Hu et al., 2021) adapted the attention projections W_q and W_v; today often all linear layers.
• QLoRA keeps the frozen base model in 4-bit precision, making fine-tuning of very large models possible on one GPU.
• Fewer trainable parameters means less capacity to memorise, but LoRA can still overfit on small datasets.

53Retrieval-augmented generation (RAG)

RAG gives an LLM access to knowledge it wasn't trained on. A retriever finds relevant passages for the question, and a reader (the LLM) answers using them. This keeps answers current, grounded in sources you control, and citable.

Design questions first

• Simplicity: what is the simplest model and pipeline that is accurate enough?
• Base vs instruction model: an instruction-tuned model expects its own chat template (role tags, special tokens); quality drops without it. A base model has no template.
• Do we also need recommendations, or only answers?
• Constraints: safety, privacy, storage, latency, cost, and how the system will be evaluated.

The two combine well: fine-tune for behaviour, retrieve for knowledge.

Chunking

• Size trade-off: small chunks are precise to retrieve but lose context; large chunks keep context but give noisier embeddings. A common start is 256–1,024 tokens with 10–20% overlap.
• Metadata (source, date, section) enables filtering and citations.
• Small-to-big / parent-document retrieval: retrieve small chunks, then pass their larger parent to the LLM.

Choosing an embedding model

• Compare on the MTEB leaderboard, then test on your own domain and languages.
• Maximum input length and vector dimension drive storage and speed.
• Use the same model for queries and documents. Some need prefixes, such as E5's “query: ” and “passage: ”.

54Retrieval: search, reranking and context

Vector stores and nearest-neighbour search

• FAISS is a library, not a database: a fast in-memory index without persistence, metadata filtering or access control.
• Vector databases: MongoDB Atlas Vector Search (convenient if your data already lives in MongoDB), pgvector, Qdrant, Weaviate, Milvus, Chroma.
• Approximate nearest-neighbour (ANN) indexes: HNSW (graph-based), IVF (clusters), PQ (compression). They trade a little recall for a lot of speed; see section 55.
• Distance metric: for unit-length vectors, cosine and dot product give the same ranking, and Euclidean too (‖a − b‖² = 2 − 2·cos). But use the metric the embedding model was trained with: for a model trained on raw dot product, normalising changes the results.

Hybrid search

Dense (vector) search captures meaning but misses exact matches: IDs, names, codes, rare terms. Keyword search (BM25) catches those. Standard practice runs both and merges the ranked lists with reciprocal rank fusion:

Query reformulation

• Query rewriting: clean up or expand the question.
• Multi-query: generate several phrasings and merge the results.
• Decomposition: split a complex question into sub-questions.
• HyDE: have the LLM write a hypothetical answer, then embed that instead of the question.

Reranking

“Learning to rank” is the broad family. In RAG the reranker is usually a cross-encoder (e.g. bge-reranker, Cohere Rerank).

Building the prompt and verifying

• Lost in the middle: LLMs use information at the start and end of the context better than the middle. Put the best chunks first or last.
• Context compression: drop irrelevant sentences to save tokens and reduce distraction.
• Verification: groundedness (faithfulness) checks, citations back to source chunks, an LLM as judge, and answering “I don't know” when nothing relevant was retrieved.

Evaluating RAG

55Vector search indexes

Retrieval needs the nearest vectors to a query among millions. There are two ways to make that faster than comparing against everything:

Flat (exact) search

Compute the distance from the query to every stored vector, with L2 or inner product (FAISS IndexFlatL2 / IndexFlatIP).

Locality-sensitive hashing (LSH)

Unlike cryptographic hashes, LSH wants collisions: similar vectors should land in the same bucket. FAISS IndexLSH projects each vector onto nbits random hyperplanes and keeps the sign of each projection as one bit; vectors are compared by Hamming distance between their codes.

• Curse of dimensionality: as d grows, nbits must grow to keep recall, and search slows down.
• Rarely the best choice for dense embeddings today; IVF and HNSW usually beat it.

Inverted file index (IVF)

k-means splits the space into nlist cells (Voronoi cells). A query searches only the nprobe cells whose centroids are nearest.

• Raising nprobe fixes the edge problem and doesn't increase memory, but it does increase search time: it trades speed for recall.
• IVF needs training (k-means on representative data). Rule of thumb: nlist ≈ 4√N to 16√N.

Product quantization (PQ)

IVF reduces how many vectors are compared; PQ makes each comparison cheap by compressing the vectors.

1. Split each d-dimensional vector into m equal subvectors.
2. Run a separate k-means on each of the m subspaces (typically 256 centroids each): the codebooks.
3. Replace each subvector with the ID of its nearest centroid.

• Asymmetric distance computation (ADC): for each query, precompute the distance from each query subvector to every centroid (an m × 256 table). A database vector's distance is then the sum of m table lookups.
• Plain PQ is still exhaustive, just cheap per vector. IVFPQ combines both levers: IVF cuts the scope, PQ compresses.
• PQ is lossy, so recall drops; rerank the top candidates with the original vectors to recover it. OPQ adds a learned rotation before PQ for better accuracy.

Hierarchical navigable small world graphs (HNSW)

In a navigable small world graph, each vector links to its near neighbours, plus some long links, so any node is reachable in a few hops. HNSW stacks several such graphs like express lanes.

• Each node gets a maximum layer drawn with exponentially decreasing probability, so higher layers hold far fewer nodes. Layer 0 holds every node.
• M: links per node (higher = better recall, more memory). efConstruction: search width while building. efSearch: search width at query time (higher = better recall, slower).
• ✓ Excellent speed and recall, no training; the default in most vector databases.
• ✗ High memory (full vectors plus every node's links); deletions are awkward.

When to use which

Figures from Pinecone's FAISS benchmark (1M vectors, 128 dimensions); they change with dataset and parameters. Every index trades off speed, memory and recall: you can't max out all three.

Optimising the embeddings themselves

• Dimensionality reduction: PCA or random projection. Matryoshka embeddings are trained so you can simply keep the first k dimensions.
• Quantisation: int8 scalar quantisation (4× smaller); binary (32× smaller), usually followed by rescoring the top results with full-precision vectors.

Infrastructure and parallelism

• GPU FAISS and batched queries.
• Sharding an index across machines; replicas for throughput.
• Metadata pre-filtering to shrink the search space; caching frequent queries.
• Disk-based indexes (DiskANN) or memory-mapped files for billion-scale data.
• Composite indexes: IVF with an HNSW coarse quantiser, IVF-PQ, HNSW-PQ.

Hybrid search (dense + BM25) is covered in section 54. For dense retrieval use sentence-embedding models (SBERT, E5, BGE): raw BERT embeddings are poor for similarity. Learned sparse retrieval (SPLADE) sits between keyword and dense search.

Further reading: Pinecone, Faiss: The Missing Manual.

56Graph RAG and multimodal RAG

Knowledge-graph RAG

A knowledge graph stores entities as nodes and relationships as edges, which helps with questions that chain facts together (“where is Alice's employer based?”).

1. Extract entities and relations from text with an LLM or NER tools (e.g. NuExtract, Haystack, or LangChain's LLMGraphTransformer; LlamaIndex's equivalent is PropertyGraphIndex).
2. Write them to the graph with Cypher, Neo4j's query language (MERGE, CREATE), which is also used for reading.
3. Query at answer time: text-to-Cypher (Lang2Cypher) has an LLM write a Cypher query from the question; the results are turned back into text (Cypher2Language) and passed to the LLM.

• Safety: generated queries can be invalid or harmful. Put the graph schema in the prompt, use a read-only connection, and validate queries before running them.
• NetworkX vs Neo4j: NetworkX builds graphs in memory (analysis, prototypes); Neo4j is a persistent graph database.
• Microsoft GraphRAG summarises clusters of related entities (“communities”), so it can answer broad questions such as “what are the main themes?” that chunk retrieval handles poorly.

Multimodal RAG with ColPali

Instead of OCR and layout parsing, ColPali retrieves page images directly from a text query. Byaldi (Answer.AI) wraps ColPali-style models in a simple API.

• Each page image is embedded by a vision-language model (PaliGemma) into one vector per image patch, about 1,000 vectors of dimension 128 per page.
• The text query gets one vector per token.
• Late interaction (MaxSim, from ColBERT): each query token finds its best-matching patch, and the matches are summed.

57Putting it together for LLMs

• Scale changes the picture. LLMs are hugely over-parameterised yet generalise well (double descent). Huge deduplicated datasets, about one epoch of pretraining, and the implicit regularisation of SGD keep variance in check.
• Typical architecture: decoder-only Transformer, pre-LN with RMSNorm, RoPE positions, GQA or MLA attention, and often mixture-of-experts feed-forward layers (Mixtral, DeepSeek-V3).
• Standard training recipe: AdamW with weight decay, linear warmup then cosine decay, gradient clipping (often max_norm = 1.0), mixed precision, and 3D/5D parallelism with ZeRO or FSDP.
• Overfitting controls: weight decay, data deduplication, dropout (often turned off in large-scale pretraining), and for fine-tuning: LoRA, low learning rates, early stopping.
• Where variance shows up: fine-tuning on small datasets (overfitting, catastrophic forgetting), verbatim memorisation of training text, and output variability from sampling temperature and prompt wording.
• Underfitting: a model too small or under-trained for the task. Fix with a larger model, more pretraining (scaling laws), or fine-tuning on the target task.

58The MLOps lifecycle

MLOps is a set of practices that streamline the whole machine-learning lifecycle, from data and development to deployment, monitoring and maintenance. It is a loop, not a line: what monitoring finds feeds back into data and retraining.

• Reproducibility runs through every stage: version data, code, dependencies and environments; use containers and automated pipelines; document every step and configuration.

59Deploying LLMs in production

Production work runs along three axes: making each request fast, building infrastructure that scales, and operating the system safely. The first axis is covered in earlier sections; this one links to them and adds the rest.

Axis 1: inference latency

• Smaller models: quantization (INT8/INT4/FP8), pruning, distillation. All are lossy: “minimal accuracy loss”, not “preserved accuracy”, so measure quality on your task. For inference, run in FP16/BF16 rather than FP32; “mixed precision” is a training term.
• Fast generation: speculative decoding and multi-token prediction, both lossless when the target verifies the draft.
• GPU optimisation: MQA/GQA shrink the KV cache (more requests fit, less memory traffic per step); FlashAttention and PagedAttention (section 50).

Batching: dynamic vs continuous

Both raise throughput, not per-token speed; per-request latency can rise slightly. See the batching figure.

Axis 2: infrastructure

Replicas are not model parallelism, and ZeRO/FSDP are training techniques that don't apply here.

• GPU choice: memory ≈ bytes per weight × parameters, plus 20–30% headroom for the KV cache. A 7B model in BF16 needs about 14 GB for weights alone.
• Cloud, on-premise or serverless: cloud for fast scaling and managed services; on-premise for data control at a higher up-front cost; serverless GPUs and managed APIs for no idle cost, with less control and cold starts.
• Autoscale on the right signal: queue depth or KV-cache usage, not CPU. Watch cold starts: loading tens of GB of weights takes minutes. Fast NVMe storage, safetensors and cached weights shorten it.
• Containers and orchestration: Docker for consistent images, Kubernetes to schedule and scale them, with serving layers such as KServe, Ray Serve, or NVIDIA Triton / NIM.
• Distributed architecture: several machines and regions remove single points of failure.

Axis 3: deployment, testing and monitoring

• Tools: Prometheus and Grafana for system metrics; LLM observability (Langfuse, LangSmith, Arize Phoenix) to trace prompts, responses and RAG steps.
• Benchmarking: load-test against SLOs (e.g. “p95 TTFT < 500 ms at 50 concurrent users”) with vLLM's benchmark scripts, NVIDIA GenAI-Perf or Locust, using realistic prompt and output lengths.
• Infrastructure as code and CI/CD: Terraform provisions infrastructure; Kubernetes manifests or Helm charts describe deployments; a model registry (MLflow, Hugging Face Hub) versions models.

Security, caching and cost

• Security and safety: authentication, rate limiting, input/output guardrails, PII filtering, prompt-injection defences.
• Caching: exact and semantic response caches, plus prefix caching.
• Cost: the smallest model that meets quality; a router that sends easy queries to a cheaper model; batch processing for offline jobs.

60Batch and real-time serving

Model serving turns a trained model into a network service that other applications can call, typically over a REST API.

Building a real-time LLM endpoint

1. Reduce latency first: model size, fast generation, GPU optimisation (section 50), e.g. serve with vLLM.
2. Serving layer: an API framework such as FastAPI.
3. Async handlers so one worker serves many users.
4. Stream tokens so users see text as it is generated.

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from openai import AsyncOpenAI          # async client: doesn't block the event loop

app, client = FastAPI(), AsyncOpenAI()

@app.post("/chat")
async def chat(msg: str):
    async def tokens():
        stream = await client.chat.completions.create(
            model="my-llm", messages=[{"role": "user", "content": msg}], stream=True)
        async for chunk in stream:
            yield f"data: {chunk.choices[0].delta.content or ''}\n\n"
    return StreamingResponse(tokens(), media_type="text/event-stream")

• Async isn't batching: while one request waits on I/O (the LLM call), the same worker serves others. Batching requests onto the GPU is the model server's job (continuous batching).
• await needs the async client; calling a sync client inside async def blocks every user. A plain def endpoint runs in FastAPI's thread pool instead.
• Streaming: one-way token streams usually use Server-Sent Events (text/event-stream), as the OpenAI API does: simple and plain HTTP. Use WebSockets for genuinely two-way connections such as voice or collaborative apps.
• Chat history: append the user's message, stream the answer, then save the full reply to the history.

Scaling up and out

• Vertical: a bigger GPU or more memory. Needed when the model doesn't fit at all.
• Horizontal: more replicas behind a load balancer. Adds throughput and removes single points of failure.
• LLMs usually need both: a big enough GPU per replica, then enough replicas.
• Microservices: split the system (UI, retrieval, model, guardrails) into services that scale independently (section 62). Data and model parallelism for training are covered in section 44.

61Model serving options

Model serving bridges development and production: it packages a trained model into a network-accessible service so other applications can request predictions, classifications or generated content.

More broadly:

For self-hosting on your own cluster, see section 62; for how vLLM works inside, see section 49.

62Serving on Kubernetes

A common path from notebook to production: wrap the model in a Python app, give it a Gradio interface, package it in a Docker container, and run it on Kubernetes (K8s), which handles deployment, scaling and recovery.

Gradio

A Python library that turns any function into an interactive web demo:

import gradio as gr
demo = gr.Interface(fn=greet, inputs=["text", "slider"], outputs=["text"])
demo.launch()

• Ideal for demos and internal tools; Streamlit is an alternative, and Hugging Face Spaces hosts them for free.
• For a public, high-traffic product, put the model behind an API server (vLLM's OpenAI-compatible server, or FastAPI) and use Gradio or a web app as a thin client.
• Gradio's queue and sessions are stateful: with several replicas, enable sticky sessions at the load balancer.

Separate the UI from the model

If the model runs inside the Gradio process, every UI pod loads its own copy onto a GPU, wasting expensive GPUs on an idle interface.

Kubernetes building blocks

• Pod: one or more containers scheduled together.
• Deployment: keeps N identical pods (replicas) running and replaces failed ones; rolling updates and rollbacks.
• Service: a stable address that load-balances across the pods.
• Ingress / Gateway: exposes the app outside the cluster, with TLS and routing.
• Containers provide most of the consistency across environments; Kubernetes adds declarative config (YAML) that keeps environments alike.

apiVersion: apps/v1
kind: Deployment
metadata: { name: llm-server }
spec:
  replicas: 2
  template:
    spec:
      containers:
      - name: vllm
        image: vllm/vllm-openai:<version>
        args: ["--model", "/models/my-llm"]
        resources:
          limits: { nvidia.com/gpu: 1 }       # needs the NVIDIA device plugin
        readinessProbe:
          httpGet: { path: /health, port: 8000 }
        volumeMounts: [{ name: weights, mountPath: /models }]

Abridged example. A Service then points at these pods, and an Ingress exposes the UI.

• GPUs: install the NVIDIA device plugin (or GPU Operator), request nvidia.com/gpu, and use node selectors or taints so GPU nodes only run GPU work.
• Model weights: mount them from a persistent volume or object storage, or download and cache at startup. Baking tens of GB into the image makes every deploy slow.
• Health checks: a startup/readiness probe keeps traffic away until the model has loaded; a liveness probe restarts a hung pod.

Autoscaling

• The default Horizontal Pod Autoscaler (HPA) scales on CPU or memory, which barely move for GPU inference.
• Scale on custom metrics instead: requests waiting, requests in flight or KV-cache usage (vLLM exposes them at /metrics), fed in through KEDA or the Prometheus Adapter.
• The Cluster Autoscaler (or Karpenter) adds GPU nodes when pods can't be scheduled. Expect minutes of cold start: node boot, image pull, model load.

Monitoring and logging

• Metrics Server only provides basic CPU/memory for kubectl top and the HPA; it isn't a monitoring system.
• Prometheus + Grafana for metrics; the DCGM exporter for GPU utilisation, memory and temperature; vLLM's own TTFT, tokens/s and KV-cache metrics.
• Loki or ELK for logs; Langfuse, LangSmith or Phoenix to trace prompts and responses (see the metrics table).

Testing in production

A typical order: shadow → canary → full rollout (see the canary figure), with A/B tests for product decisions.

63CI/CD for ML

Continuous integration (CI) tests every change automatically; continuous deployment (CD) ships changes that pass. For ML, “every change” includes code, data and models, so the pipeline also trains, evaluates and versions models.

Triggers

Triggering CI only on push to main means a bad change is already merged before anyone sees the results.

Building the CI pipeline

• Makefile: one set of commands (make install, make train, make eval) for both laptops and CI, so “works on my machine” problems surface early. Useful, not required.
• Environment: pick a runner (ubuntu-latest), set up Python, pin dependency versions and cache installs. Hosted runners have no GPU: use a self-hosted GPU runner or have CI launch a cloud or Kubernetes job (CML can provision cloud runners).
• CML (Continuous Machine Learning, by Iterative) posts metrics, plots and confusion matrices as a comment on the pull request (cml comment create report.md). Any email comes from GitHub notifications.

name: ci
on:
  pull_request:
  push: { branches: [main] }
jobs:
  train-eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.11", cache: pip }
      - uses: iterative/setup-cml@v2
      - run: make install lint test
      - run: make train eval          # writes metrics.json + report.md
      - run: python check_gate.py     # exit 1 if worse than baseline
      - env: { REPO_TOKEN: "${{ secrets.GITHUB_TOKEN }}" }
        run: cml comment create report.md

Abridged example; action versions are illustrative.

ML-specific checks

Versioning models and data

• Don't commit model files to git: they bloat the repository, and GitHub rejects files over 100 MB.
• Don't let CI push straight to main: it skips review and can retrigger the workflow in a loop. If CI must change files, open a pull request from a bot branch.
• Version data and models with DVC, Git LFS, the MLflow model registry or the Hugging Face Hub. Tag the code commit (v1.3.0) and record the matching model version.

Building the CD pipeline

1. Fetch the trained model (CI artifact or registry).
2. Upload it to the model registry or Hugging Face Hub.
3. Build and push a Docker image.
4. Deploy to Hugging Face Spaces (Gradio) or Kubernetes, releasing via shadow → canary → full rollout with automatic rollback (section 59).

• Authentication: store tokens such as HF_TOKEN as GitHub Secrets, never in code; for cloud providers prefer OIDC to long-lived keys.

MLOps maturity levels

ML needs continuous training because a model can degrade when the data changes, even if the code doesn't (Google Cloud, MLOps: continuous delivery and automation pipelines in machine learning). Further reading: DataCamp, CI/CD for Machine Learning.

64Drift and model monitoring

A deployed model can get worse without any code change, because the world it sees changes. Labelled “golden” data is rarely available in production, so monitoring relies on proxy signals.

• Embedding-based detection: compare embeddings of current inputs and outputs with a baseline; a falling similarity suggests a shift in meaning. For full embeddings, use multivariate tests (MMD) or a classifier trained to tell old inputs from new.
• Statistical tests: the KS test compares one numeric feature at a time (e.g. prompt length) against a baseline.
• Perplexity: rising perplexity on new inputs means they are unlike the training data (data drift). It doesn't directly measure answer quality.
• LLM-as-judge scores thousands of outputs quickly; calibrate it against a sample of human labels first, since judges have biases such as favouring longer answers.
• User feedback: explicit ratings (thumbs up/down) and implicit signals (edits, retries, abandoned sessions).
• Human in the loop: experts periodically review a sample of outputs.

Monitoring tools

65Versioning, rollbacks and governance

Model versioning

Track, reproduce and roll back models safely as they evolve. Version every artifact of the pipeline:

• Model weights, training data version, preprocessing and training code, hyperparameters and configuration, metrics, and the deployment environment.
• Prompts and RAG indexes too: a prompt change can break behaviour as badly as a model change.
• A model registry (e.g. MLflow) stores versions; aliases such as “champion” and “challenger” point at them, so every deployment uses a specific registered version.
• Containers and CI/CD make each version reproducible and tested.

Rollbacks

• Revert to the previous stable version when a new one causes errors or degradation; with aliases, rolling back means moving the alias back.
• Keep the previous version warm (blue-green) for instant switching; trigger rollback automatically on metric breaches (canary figure).
• Check the new version is compatible with the current data schema, so rolling back doesn't break consumers.

Sensitive data

• Encrypt data at rest and in transit; strict access control; anonymise or pseudonymise personal identifiers.
• Redact PII before sending prompts to third-party APIs, and check the provider's data-retention terms.
• Logs are sensitive: stored prompts and responses contain user data and need retention limits and access control.
• Right to erasure (GDPR): removing a person's data from a trained model is hard; keep personal data out of training, or be ready to retrain.
• Regulations: GDPR, HIPAA, and the EU AI Act for high-risk uses.

Interpretability vs explainability

The terms are often used loosely, so define them when you use them.

66Model Context Protocol (MCP)

MCP is an open protocol that standardises how AI applications connect to tools and data. Instead of a custom connector for every pair of app and system, each app implements an MCP client and each system an MCP server.

Host, client and server

• Host: the AI application (an IDE or chat app). It runs the LLM and decides what to do with tool results.
• Client: a component inside the host, with one client per server (a 1:1 connection). A host connected to three servers runs three clients.
• Server: a program that exposes capabilities and executes them (an API call, a database query).
• Protocol: messages are JSON-RPC 2.0. A session starts with an initialize handshake where both sides agree on capabilities; the client then calls tools/list to discover tools and tools/call to run one.

Transports

A tool call, step by step

Server primitives

Servers can also ask things of the client: sampling (borrow the host's LLM) and elicitation (ask the user for input).

MCP vs function calling

• Function calling is a model-API feature: the model outputs a structured call to a function you described.
• MCP standardises how tools are discovered, described and served across apps. MCP tools usually reach the model through function calling.

One server or many

Too many tools hurts the model, whatever the number of servers: every tool description takes context, and with dozens of tools the LLM picks the wrong one more often. Expose only the tools a task needs, write clear and distinct descriptions, or select tools dynamically.

Security

• Prompt injection through tool results: a web page or email returned by a tool can contain instructions aimed at the LLM.
• Tool poisoning: a malicious server hides instructions in its tool descriptions.
• Mitigations: install only trusted servers; give each least privilege (read-only where possible); require user approval for sensitive actions; use OAuth for remote HTTP servers; log every tool call (section 65).

Further reading: modelcontextprotocol.io, the specification and documentation.

67Agent memory

An LLM forgets everything between calls. Agents add memory around it: short-term memory holds the immediate context of one interaction; long-term memory persists knowledge across sessions, tasks and time. The taxonomy follows cognitive science and the CoALA framework (Sumers et al., 2023).

Short-term (working) memory

The context window itself: system prompt, available tools, retrieved context, and the reasoning-and-action history (scratchpad), assembled by the orchestrator for each call.

• Limit: context length and cost. As a conversation grows, trim old turns or keep a sliding window, summarise earlier turns, or move details to long-term memory and retrieve them when needed.

Long-term memory

• Episodic: after each action, the application stores what happened. Retrieving similar past episodes and showing them as examples of what worked is a common few-shot technique.
• Semantic: a private knowledge base plus grounding context is exactly RAG. Live web search is a tool, not memory.
• Procedural: the agent's “how-to”: update it by editing prompts and tools, or by fine-tuning, which changes what the model itself knows how to do.

Writing and managing memory

• Hot path: the agent calls a save_memory tool during the conversation. Memory is immediate, but each turn is slower.
• Background: a separate job extracts memories after the conversation. Chat stays fast; memory lags.
• Consolidation: an LLM “reflection” step turns episodes into semantic facts, e.g. three conversations about Python errors become “the user codes in Python”.
• Updating and forgetting: new facts can conflict with old ones (“moved to Paris”), so memories need update and delete, not just append; unused memories should decay or be pruned.

Retrieving the right memories

Relevance alone isn't enough. Generative Agents (Park et al., 2023) combines three signals:

Risks

• Stale or wrong memories keep misleading the agent until corrected.
• Memory poisoning: a prompt injection that gets saved affects every future session. Validate what gets written (section 66).
• Privacy: memories are personal data. Let users view and delete them; the right to erasure applies (section 65).
• Cost: every retrieval adds latency and context tokens.

Tools

• LangGraph: checkpoints for short-term memory, a store for long-term memory.
• Mem0; Zep (a temporal knowledge graph that tracks when facts were true).
• Letta (formerly MemGPT): manages memory like an operating system, paging information between the context window and external storage.

Further reading: Sumers et al., Cognitive Architectures for Language Agents (2023); Park et al., Generative Agents (2023).

68Check yourself

Try to answer before opening each one.