Running Models Locally.

You run Linux servers for a living, so the OS is not the interesting axis — the memory architecture and compute API are. Local inference lives in one of three worlds. The production baseline is Linux (or Windows) with an NVIDIA GPU: the stack is CUDACUDANVIDIA's parallel programming model and runtime for writing GPU kernels. → driver → a discrete GPU with its own HBMHBMHigh-Bandwidth Memory — the stacked DRAM on a datacenter GPU (the 'VRAM')./GDDR. Two physical memories exist — host RAM (DDR) and device VRAMVRAMThe GPU's dedicated memory; must hold weights + KV cache + activations. — joined by PCIePCIeThe general-purpose bus linking GPU, CPU and NIC; the slow path for GPU-GPU traffic. at ~32 GB/s, roughly 30× slower than VRAM. The rule that follows recurs in every later module: load weights to VRAM once, keep them resident, never stream per-token across PCIe.

The second world is Apple Silicon, which has no separate VRAM. CPU cores, GPU cores and the Neural Engine share one physical pool of LPDDR5 on-package over a very wide bus — unified memoryUnified memoryOne physical memory pool shared by CPU and GPU — as on Apple Silicon.. The compute API is Metal (PyTorch exposes it as MPS; llama.cpp has a native Metal backend). Bandwidth is the catch: ~400 GB/s on an M3 Max, ~800 GB/s on an M-Ultra — respectable, but well below an H100's ~3.35 TB/s. A Mac wins on capacity and loses on throughput.

The third is CPU-only fallback. Every engine has a CPU path (llama.cpp shines here with AVX2/AVX-512/NEON and GGUF), and it always works — it is just bandwidth-starved: dual-channel DDR5 is ~50–90 GB/s versus a GPU's 1,000–3,350 GB/s. Since decode is bandwidth-bound, CPU decode is roughly 10–40× slower for the same model. Fine for a small, latency-tolerant model; painful for interactive large ones. Windows+NVIDIA is architecturally identical to Linux+NVIDIA — same CUDA, same discrete split — so the memory math never changes; that is why we treat them as one world.

Four tools cover essentially all of local and self-hosted inference, and they split cleanly along one axis: single-user local convenience versus multi-user server throughput. llama.cpp is the C/C++ reference implementation (with ggml as its tensor library). It has zero heavy dependencies, runs everywhere (CUDA, Metal, Vulkan, ROCm, pure CPU), and invented both the GGUFGGUFllama.cpp's single-file model format with metadata + quantized tensors, memory-mapped at load. format and the K-quant scheme. It ships llama-server (OpenAI-compatible) and llama-cli, and it is optimized for low-batch, single-user latency. Ollama is a batteries-included wrapper built on llama.cpp — the "Docker for models": ollama run llama3 pulls a curated GGUF, picks a quant, sets GPU layers and serves an API. It inherits llama.cpp's single-user profile.

vLLM is a Python/CUDA inference server built for throughput and concurrency. Its signature contribution is PagedAttention — a virtual-memory-like KV-cacheKV cacheStored key/value vectors for all past tokens, so each new token needs only one forward pass. allocator that eliminates fragmentation and enables large batches — plus continuous batching (new requests join the running batch every step). It consumes SafetensorsSafetensorsA safe, zero-copy tensor file format that replaced Python pickle for model weights. in the common path, is NVIDIA-first, and is the default for self-hosting real traffic (internals are Module 4). TensorRT-LLM is NVIDIA's maximum-performance path: it compiles a model into a hardware-specific TensorRT engine — kernels fused and tuned for one exact GPU (SMStreaming Multiprocessor (SM)The GPU's core compute unit — holds CUDA cores, Tensor Cores, registers and shared memory. version), with FP8/INT4 and in-flight batching — for the highest tokens/sec on NVIDIA, at the cost of a build step and much less flexibility (an engine built for H100 will not run on an A100).

The rule of thumb: Mac or laptop → Ollama/llama.cpp. One NVIDIA GPU serving users → vLLM. A fleet of identical GPUs, latency-critical → TensorRT-LLM. The comparison table below sorts these along the axes you actually decide on: what each is built on, its concurrency profile, the formats it eats, and the platforms it supports.

A model on disk is just a large collection of named tensors plus metadata. The format decides how fast you can load them, whether you can mmap them, and — crucially — whether reading the file can execute code. The cautionary tale is pickle: PyTorch's original .bin/.pt checkpoints are Python pickle streams, and unpickling can execute arbitrary code (__reduce__), so loading a random model off the internet was literally remote code execution. Both modern formats exist to kill pickle.

SafetensorsSafetensorsA safe, zero-copy tensor file format that replaced Python pickle for model weights. is Hugging Face's format, now the default for full-precision (fp16/bf16) weights. The layout is dead simple: an 8-byte header length, a JSON header describing each tensor (name, dtype, shape, byte offsets), then raw tensor bytes back-to-back. Two properties matter. It is safe — pure data, parsing cannot execute code — and it is zero-copy: because tensors are contiguous bytes at known offsets, a loader can mmap the file and hand slices straight to the framework (or cudaMemcpy to GPU) with no deserialization pass, so loading is bounded by disk/PCIe bandwidth, not CPU. It is what vLLM and TensorRT-LLM ingest.

GGUFGGUFllama.cpp's single-file model format with metadata + quantized tensors, memory-mapped at load. is llama.cpp's format, purpose-built for quantized, CPU/GPU-portable inference. Unlike Safetensors it is self-contained: one file holds the tensors plus everything needed to run — architecture, hyperparameters, the tokenizer (vocab + merges), and the chat template — so you need no separate config.json. It is mmap-friendly with lazy fault-in: the engine maps the file and the OS pages tensors in on demand, so the process starts before the whole file is read, two processes mapping the same GGUF share physical pages, and those file-backed pages are reclaimable under pressure. On a GPU run the mapped file is only the staging copy; the resident copy lives in VRAM. Mental model: Safetensors is the interchange format for full-precision weights; GGUF is the deployment format for quantized local inference.

QuantizationQuantizationStoring weights (and sometimes activations/KV) in fewer bits to cut memory and bandwidth. means storing weights in fewer bits than the fp16 they were trained in. Because decode is bandwidth-bound (fewer bytes per weight = fewer bytes to stream per token = faster) and it shrinks the resident footprint, it is the single highest-leverage knob for local inference — Module 1 told you decode re-reads all weights every token, and this attacks that cost directly. fp16 is 16 bits/weight; "Q4" is ~4 bits/weight, a ~4× shrink. But the real numbers are not exactly 4.0, because quantization works in blocks (~32 weights) with a shared scale (and sometimes a zero-point) per block, and those scales cost bits too, so the effective bits-per-weight sits a little above the nominal number.

Decoding a name like Q4_K_M: Q4 is the nominal ~4 bits/weight; _K is the K-quant superblock scheme (sub-blocks with hierarchical scales), higher quality than the legacy Q4_0/Q4_1 round-to-nearest quants; and _S/_M/_L is a Small/Medium/Large mix — sensitive tensors (attention output, feed-forward down-projections, often embeddings) are kept at higher precision while the bulk stays at the nominal level, and _M keeps more of them high than _S. This mixed-precision-by-importance is why K-quants punch above their bit count. What gets quantized is the big weight matrices (Q/K/V/O and MLP up/gate/down — ~99% of parameters); layer-norms and often the embedding/output matrices are kept higher. Activations and the KV cache are a separate runtime story (Module 9).

Quality is usually reported as perplexity increase versus the fp16 baseline. The sortable table below gives approximate, community-consensus figures for a ~7–8B model — order-of-magnitude, not benchmarks. Two rules hold in practice. Q4_K_M is the default — the knee of the curve, ~4× smaller than fp16 with loss most users cannot feel in chat. And a bigger model at a lower quant usually beats a smaller model at a higher quant at equal file size: a 70B at Q2_K (~26 GB) generally out-reasons a 13B at Q8 (~14 GB). Spend your bytes on parameters first, precision second — down to about Q3; below that the degradation gets steep.

This is the crux of the module: when you "load a model," several distinct things take up memory, and conflating them is the number-one source of "why did I OOM?" confusion. There are four residents. Weights — the parameters, static, sized by $\text{params}\times \text{bits}/8$, the big one. KV cacheKV cacheStored key/value vectors for all past tokens, so each new token needs only one forward pass. — per-token, per-layer K and V vectors cached during decode, which grows with context length and batch size (the one people forget). Activations — transient tensors for the current forward pass, small in decode and larger in prefill. And context/compute buffers — CUDA/Metal context, kernel scratch, allocator overhead: a roughly fixed tax of hundreds of MB to ~1–2 GB.

Two formulas do all the work. Weights are $\text{params}\times (\text{bits/weight}/8)$. The KV cache is $2\cdot n_{layers}\cdot n_{kv\text{-}heads}\cdot d_{head}\cdot \text{seq}\cdot \text{batch}\cdot \text{dtype}$ — note $n_{kv\text{-}heads}$, not $n_{heads}$: with GQAGrouped-query attention (GQA)Multiple query heads share a smaller number of key/value heads, shrinking the KV cache., Llama-3-8B has 32 query heads but only 8 KV heads, a 4× cut in KV size. Worked example — Llama-3-8B as GGUF Q4_K_M at 8k context, batch 1, KV at fp16: weights $\approx 8.03\text{e}9\times 4.8/8\approx 4.82$ GB; KV $\approx 2\times 32\times 8\times 128\times 8192\times 2\approx 1.07$ GB; activations ~0.1 GB; buffers ~0.4 GB — total ~6.3 GB, comfortably inside a 24 GB GPU. Once weights are copied to VRAM, the mmap'd file in host RAM is just a reclaimable staging copy; the GPU never touches disk again during generation, and the only thing that grows while the user chats is the KV cache.

The same model at fp16 (the Safetensors/vLLM path) tells the other half of the story: weights $\approx 8.03\text{e}9\times 2\approx 16.06$ GB, plus ~1 GB KV and ~2 GB of CUDA/PyTorch context — ~18–20 GB, leaving only ~5 GB free on a 24 GB card versus ~17 GB free for Q4_K_M. That headroom is your concurrency and context budget, which is exactly why production servers quantize even when the fp16 weights "fit." Toggle the residency diagram below between a discrete GPU and an Apple unified pool, and use the VRAM budget calculator to watch weights + KV + overhead compete for the same scarce memory.

Here is the architectural point that surprises people, made precise. On a discrete GPU (say an RTX 4090, 24 GB) the weights must fit in the GPU's 24 GB, or you offload layers to host RAM and pay PCIePCIeThe general-purpose bus linking GPU, CPU and NIC; the slow path for GPU-GPU traffic. per token. A fp16 70B is ~140 GB — it simply does not fit — and even a Q4 70B (~40 GB) does not fit in 24 GB. You are capped hard by the number printed on the box.

On a unified-memoryUnified memoryOne physical memory pool shared by CPU and GPU — as on Apple Silicon. Mac (128 GB) the GPU cores can address the entire pool. A Q4 70B (~40 GB of weights plus KV) fits with room to spare, and the GPU is not reading over PCIe — the LPDDR5 is its memory, so there is no offload and no per-token host round-trip. The tradeoff is bandwidth, and it maps straight onto Module 1's bandwidth-bound decode: a 70B Q4 (~40 GB streamed per token) at ~400 GB/s tops out around ~10 tok/s on an M-series — usable for a single reader, not for serving. A 4090 would do far more tok/s on any model that fits its 24 GB, but cannot hold the 70B at all.

So: Mac = capacity king; NVIDIA = throughput king. Choose by whether your constraint is "can it fit" or "how many tokens/sec and how many users." This is also why the datacenter answer is not "buy a huge unified box" — it is to pool many high-bandwidth GPUs over NVLinkNVLink / NVSwitchHigh-bandwidth direct GPU-to-GPU interconnect within a server (far faster than PCIe)., getting both capacity and bandwidth (Modules 5 and 6). The VRAM budget calculator lets you sweep model, precision and GPU to see the fit boundary move.

The instinct from CPU-land is "GPUs have more cores/FLOPs." True, but for decode that is not why they win. Decode is memory-bandwidth-boundMemory-boundLimited by memory bandwidth, not compute — the GPU waits on HBM, not the math units.: each token requires reading every weight once, and the arithmetic per byte read is low (a matrix–vector product), so the ALUs sit idle waiting on memory. Throughput is therefore approximately $\text{memory bandwidth}/\text{model bytes}$. Plug in Llama-3-8B Q4_K_M (~4.8 GB read per token) and the ceiling scales almost linearly with bandwidth — that is the whole story, and why every serious deployment is on high-bandwidth HBM. The table below makes the linearity concrete across DDR5, an M3 Max, a 4090 and an H100.

Prefill is the exception: it is compute-bound because it processes many tokens in parallel, so GPUs win there on FLOPs too (Module 1's prefill/decode split). CPU is acceptable for small models (≤3–8B), latency-tolerant or batch workloads, no GPU available, or dev/testing — GGUF + llama.cpp on CPU is genuinely fine for a background 3B doing occasional summarization. For anything interactive or multi-user, use a GPU.

Partial offload (llama.cpp --n-gpu-layers) is the hybrid case: put as many layers as fit in VRAM and the rest in CPU RAM. Every token, the CPU-resident layers are computed on CPU (or their weights streamed over PCIe), so effective throughput is dragged toward the slow tier — offloading even ~20% of layers can halve tok/s. Useful to run a model slightly too big for your VRAM, but the cliff is steep: prefer a smaller quant that fully fits over offloading. The repeatable back-of-envelope — weight_bytes, kv_bytes, overhead, fit?, and a bandwidth-derived speed ceiling — is in the code below.