How LLM Inference Actually Works.

When you send a prompt, no text ever reaches the neural network. The string is first turned into a list of integer IDs by the TokenizerTokenizerDeterministic algorithm mapping text ↔ token IDs, usually byte-level BPE., those IDs index into a table of learned vectors, and from then on the model manipulates only numbers. What comes back is, one at a time, the single most-likely next tokenTokenThe atomic unit a model reads and generates — a subword piece, not a word or character. — sampled, appended to the sequence, and fed back in. Generation is a loop, not a function call.

At the systems level the path is: text → token IDs → embedding vectors → N Transformer layers → LogitsLogitsRaw pre-softmax scores over the vocabulary for the next token. over the vocabulary → sample one token → repeat. Everything expensive happens in those N layers, and almost all of the interesting engineering later in this course exists to make that loop cheaper, more parallel, or more memory-efficient.

The single most important structural fact: the work divides into two phases with completely different performance characteristics. Processing the prompt (PrefillPrefillThe compute-bound first phase: process the whole prompt in parallel, fill the KV cache.) happens once, in parallel, and is limited by compute. Generating the answer (DecodeDecodeThe memory-bound second phase: generate one token at a time, autoregressively.) happens token by token, is inherently sequential, and is limited by memory bandwidth. Confuse these two and every capacity estimate you make will be wrong.

A TokenTokenThe atomic unit a model reads and generates — a subword piece, not a word or character. is a subword unit, not a word or a character. Modern models use byte-level BPE (byte-pair encoding): starting from raw UTF-8 bytes, the tokenizer greedily merges the most frequent adjacent pairs into a fixed vocabulary of typically 32k–256k entries. Common words become a single token; rare words and code fragment into several. A useful rule of thumb is ~1 token ≈ 0.75 English words.

Byte-level BPE has one property that matters operationally: it can represent any input — there is no out-of-vocabulary case, because the fallback is always individual bytes. Emoji, Base64, a language the model barely saw in training — all encode. This is why your token counts (and therefore your costs and your context-window usage) are only loosely correlated with character counts.

Tokenization is deterministic, cheap, and CPU-side — it is never the bottleneck. But it shapes everything downstream: the vocabulary size sets the width of the embedding table and the final projection, and the token count sets how much KV cacheKV cacheStored key/value vectors for all past tokens, so each new token needs only one forward pass. you will allocate.

Each token ID is converted to a vector by a pure table lookup into the embedding matrix, which has shape $[\text{vocab},d_{model}]$ — no matrix multiply, just picking a row. That $d_{model}$-dimensional vector (e.g. 4096 for Llama-2-7B) is the token's position in the residual stream: the running hidden state that flows through every layer and that each layer reads from and adds back to.

The width $d_{model}$ is the single most important shape in the model. Almost every weight matrix is $O(d_{model}^2)$, so it dominates both parameter count and per-token compute. When you later reason about why a 70B model is ~140 GB in FP16, this is the lever: parameters scale with the square of the hidden size times the number of layers.

Because attention is order-blind, positional information is injected here too (via rotary embeddings, RoPE, in most modern models) so that the model can tell token 5 from token 50. From this point on, the sequence is a matrix of shape $[\text{seq\_len},d_{model}]$ moving through the stack.

Inside each layer, AttentionAttentionMechanism letting each token read a weighted mix of other tokens' values. lets every token read a weighted blend of other tokens. Each token produces a query ($Q$), and every token exposes a key ($K$) and a value ($V$). The output for a token is $\text{softmax}(Q{K}^{\top }/\sqrt{d_k})V$: score your query against all keys, normalize, and gather a weighted sum of values. In a decoder this is causal — a token may only attend to itself and earlier positions.

Here is the insight that makes generation tractable. During decoding, the $K$ and $V$ vectors of every past token never change. Recomputing them for the whole history at every step would be quadratic and ruinous. So we cache them: the KV cacheKV cacheStored key/value vectors for all past tokens, so each new token needs only one forward pass. stores all past keys and values in HBM, and each new token computes just its own $Q$, attends against the cached $K$/$V$, and appends its own $K$/$V$. Generation becomes linear per step instead of quadratic.

That cache is not free. Its size per token is $2\cdot n_{layers}\cdot n_{kv\text{-}heads}\cdot d_{head}\cdot \text{dtype}$ bytes — the 2 is one key plus one value — and it grows linearly with sequence length and with batch size. It is the dominant runtime memory consumer and, as the next sections show, the main reason decode is memory-bound. Modern models fight its growth with grouped-query attentionGrouped-query attention (GQA)Multiple query heads share a smaller number of key/value heads, shrinking the KV cache., where many query heads share a few KV heads.

Prefill processes your entire prompt in a single forward pass. Because all $N$ prompt tokens go through together, the core operations are big matrix–matrix multiplies that saturate the GPU's Tensor CoresTensor CoreSpecialized unit doing small matrix multiply-accumulates — the workhorse for LLMs.: prefill is compute-bound, and its duration sets your time-to-first-tokenTTFT (time to first token)Latency from request to the first generated token — dominated by prefill.. Longer prompts cost more prefill, roughly linearly (attention adds a quadratic term, but the matmuls dominate at typical lengths).

Decode then generates the answer one token at a time. Each step is a matrix–vector multiply: a single token's activations against the full weight matrices, plus a read of the entire KV cacheKV cacheStored key/value vectors for all past tokens, so each new token needs only one forward pass.. Because token $t+1$ cannot begin until token $t$ exists, decode is inherently sequential — you cannot parallelize across time within one sequence. This is why a 500-token answer takes ~500 sequential steps no matter how much compute you throw at it.

The play button below traces the loop: watch the prompt light up all at once (prefill), then tokens emit one by one (decode). Almost every optimization later in this course — continuous batching, paged KV cache, speculative decoding — is an attack on the cost or the sequentiality of that decode loop.

To generate one token, the GPU must read every weight in the model from HBM at least once, plus the entire KV cache — and with all that traffic it produces a single token's worth of math. The ratio of useful arithmetic to bytes moved, the arithmetic intensityArithmetic intensityFLOPs performed per byte moved from memory — decides compute- vs memory-bound., is only ~1–2 FLOP per byte. A modern GPU wants a few hundred FLOP/byte before its compute units are the bottleneck. Decode lives deep in the Memory-boundMemory-boundLimited by memory bandwidth, not compute — the GPU waits on HBM, not the math units. region — often using single-digit percentages of the GPU's peak FLOPs.

This flips the usual intuition. For single-stream decode, tokens-per-second is governed by HBM bandwidth, not FLOPs: roughly $\text{TPOT}\approx \text{bytes read per token}/\text{bandwidth}$. A 70B FP16 model touches ~140 GB per token; on an H100 at ~3.35 TB/s that is a hard floor near ~42 ms/token (~24 tok/s) before any overhead. The roofline below makes the gap between decode and the compute roof visual — and lets you see that a faster GPU raises the roof but does not move decode out of the memory-bound zone.

The escape hatch is batching: run many sequences at once so each expensive weight read is amortized across dozens of tokens. That raises effective arithmetic intensity and pushes you toward the compute roof — trading a little per-request latency for a large gain in aggregate Throughput vs latencyThroughput vs latencySystem-wide tokens/sec across all requests vs the speed felt by one request.. This single fact — decode is memory-bound, batching fixes it — is the economic foundation of every serving system in Modules 4–9.

The final layer projects the hidden state to a vector of LogitsLogitsRaw pre-softmax scores over the vocabulary for the next token. — one raw score per vocabulary entry. SamplingSamplingChoosing the next token from the logits — greedy, temperature, top-k or top-p. turns those scores into an actual token. Greedy decoding takes the argmax (fully deterministic). Temperature divides the logits before softmax: below 1 sharpens toward the top choice, above 1 flattens toward randomness.

On top of temperature, two truncation strategies keep sampling from picking absurd low-probability tokens. Top-k keeps only the $k$ highest-probability tokens; top-p (nucleus) keeps the smallest set whose probabilities sum to $p$ (e.g. 0.9), adapting the cutoff to how confident the model is. These are cheap CPU/GPU post-processing steps — negligible next to a forward pass — but they define the character of the output.

Sampling is also where features like logit bias, repetition penalties, and structured/JSON-constrained decoding hook in, by masking or reshaping the logits before the draw. Nothing here changes the performance story: decode is still one forward pass per token; sampling just decides which token that pass yields.

Three numbers describe inference latency, and they map directly onto the two phases. TTFTTTFT (time to first token)Latency from request to the first generated token — dominated by prefill. (time-to-first-token) is how long the user waits before text starts appearing — dominated by queueing plus prefill, and it grows with prompt length. TPOTTPOT / ITLTime per output token (a.k.a. inter-token latency) — the streaming speed after the first token. (time-per-output-token, a.k.a. inter-token latency) is the streaming speed once generation begins — governed by decode and therefore by HBM bandwidth.

TPS (tokens per second) comes in two flavors that are constantly confused. Per-stream TPS is just $1/\text{TPOT}$ — the speed one user feels. System TPS is the total across all concurrent requests, which batching drives far higher than any single stream. A server can deliver 20 tok/s to each of 100 users (system TPS 2000) even though no single user exceeds 20.

This is the throughput-vs-latencyThroughput vs latencySystem-wide tokens/sec across all requests vs the speed felt by one request. tension that defines serving. You tune batch size, scheduling, and the prefill/decode mix to maximize system TPS while keeping TTFT and TPOT inside an SLA. Total user-visible latency is simply $\text{TTFT}+(\text{output tokens}-1)\times \text{TPOT}$ — the timeline widget below breaks it apart.