← LLM Complete GuideLayer 0
Prerequisites — the math & ML floor.
You don't need a PhD — you need working intuition for a handful of ideas that keep reappearing. A transformer is, at its core, a stack of matrix multiplications trained by gradient descent. Get comfortable with the math underneath that sentence and everything later becomes concrete instead of magic.
4
math areas
algebra · calc · prob · info
1
framework
PyTorch
build 1
tiny model
nanoGPT / makemore
~90%
of a transformer
is matrix multiplies
What to learn
The concepts this layer is made of. Aim for working intuition on each — you'll deepen them as you build.
1Linear algebra
Vectors, matrices, matrix multiplication, dot products, and the idea of a vector space. This is ~90% of what a transformer computes — attention and every layer are matmuls. You need to feel what a dot product measures (similarity) and how a matrix transforms a vector.
2Calculus
Derivatives, gradients, and the chain rule. The chain rule is backpropagation — the algorithm that trains every model. You don't need to compute integrals; you need to understand how a gradient points toward lower loss.
3Probability & statistics
Probability distributions, the softmax function, cross-entropy loss, maximum likelihood, and perplexity. An LLM outputs a probability distribution over the next token — this is the language it speaks.
4Machine learning basics
The difference between training and inference, what a loss function is, gradient descent, overfitting vs generalization, and the train / validation / test split. These framings apply to every model, LLMs included.
5Deep learning basics
Neural networks (layers of weights + activations), common activations (ReLU, GELU), embeddings, optimizers (SGD, Adam/AdamW), and backpropagation. This is the bridge from ML basics to transformers.
6Tooling — Python + PyTorch
Python, then PyTorch specifically: tensors, autograd (automatic differentiation), and how a training loop looks in code. Building a tiny model yourself teaches more than any amount of reading — Karpathy's makemore and nanoGPT are the fastest path.
Roadmap — the order to learn it
Watch 3Blue1Brown's Essence of Linear Algebra — get the visual intuition for vectors, matrices, and dot products before any code.
Watch 3Blue1Brown's Neural Networks series — it connects the math to what a network actually does, including a gentle backprop explanation.
Learn just enough Python + PyTorch tensors and autograd to write a training loop (PyTorch's 60-minute blitz).
Do Karpathy's Neural Networks: Zero to Hero — you build backprop, then
makemore, then a GPT, from scratch. This collapses everything above into concrete understanding.Optional depth: skim the relevant chapters of Dive into Deep Learning (d2l.ai) or fast.ai for anything that still feels fuzzy.
You're ready to move on when…
- You can explain what a dot product measures and why attention uses it.
- You can describe, in one paragraph, how gradient descent + backprop train a network.
- You know why a model outputs a probability distribution and what cross-entropy penalizes.
- You have trained some tiny model in PyTorch and watched the loss go down.
Resources
A curated shortlist — quality over quantity. Start at the top.