Interactive Paper Library

Course3 of 9 modules live

LLM Deployment,
from first principles.

From your laptop to hyperscale H100 clusters — how large language models are actually served, built for engineers who already know distributed systems.

Written for engineers fluent in Java, Kubernetes, AWS, Docker, distributed systems and networking. No hand-holding on those — deep on everything GPU- and inference-specific.

Start with Module 1 →

How the stack fits together

A request flows top-down; each layer is a module. Every optimization exists to make the bottom two layers cheaper at the top.

Production & cloud

gateway → auth → LB → scheduler → GPU cluster · AWS / GCP

7. Production8. Cloud

The inference engine

continuous batching · paged KV cache · prefix caching · schedulers

4. Inference engines

Scaling across GPUs & nodes

tensor / pipeline / expert parallelism · NCCL · NVLink · RDMA

5. Multi-GPU6. Multi-node

Hardware & optimization

SMs · Tensor Cores · HBM · the roofline · quantization · FlashAttention

3. GPU architecture9. Performance

Foundations

how inference works · running models locally

1. Inference internals2. Running locally

The nine modules

Available

How LLM Inference Actually Works

What happens between your prompt and the first token — tokenization, attention, the KV cache, and why generation is sequential and memory-bound.

2 phases

prefill vs decode

~1 FLOP/B

decode intensity

TTFT/TPOT

the metrics

Available

Running Models Locally

llama.cpp, Ollama, vLLM and TensorRT-LLM across Windows, Linux and Apple Silicon — GGUF, quantization, and exactly what loads into RAM vs VRAM.

Q2–Q8

quant levels

unified

vs discrete VRAM

byte map

of the load

Available

GPU Architecture

CUDA, SMs, Tensor Cores, the HBM memory hierarchy, NVLink and MIG — why the H100 is fast, and how 4090 / H100 / B200 / GB200 compare.

~3.35 TB/s

H100 HBM3

~900 GB/s

NVLink 4

4 GPUs

compared

Coming soon

Inference Engines, Internally

Continuous batching, PagedAttention, prefix caching, CUDA graphs and speculative decoding — the algorithms and data structures inside a serving engine.

continuous

batching

paged

KV cache

7 engines

compared

Coming soon

Multi-GPU Inference

Tensor / pipeline / expert parallelism, NCCL collectives, and a full trace of one token generated across eight H100 GPUs.

TP/PP/EP

strategies

160

AllReduces/token

8×H100

worked example

Coming soon

Multi-Node Inference

Crossing the node boundary: RDMA, RoCE and InfiniBand, scaling 2 → 64 → 1024 GPUs, fault tolerance and KV-aware load balancing.

RDMA

kernel bypass

1024

GPUs

N+1

fault tolerance

Coming soon

Production Deployment

The full request lifecycle, prefill/decode disaggregation, GPU-aware Kubernetes, and representative ChatGPT / Claude / DeepSeek serving architectures.

gateway→GPU

lifecycle

disaggregated

prefill/decode

3 stacks

reference

Coming soon

Cloud Deployment

AWS vs GCP for inference: GPU instances, EFA / GPUDirect networking, EKS / GKE, spot economics and managed inference services.

P5 vs A3

instances

spot

economics

build/buy

framework

Coming soon

Performance Optimization

The optimization toolbox on one roofline: quantization, FlashAttention, KV compression, speculative decoding — and when each one pays off.

roofline

framing

FP16→INT4

quant

playbook

prioritized