← LLM Complete GuideLayer 1
Sequence modeling — why transformers exist.
Before the Transformer, understand the problem it solves. An LLM is a sequence model trained on one deceptively simple task: predict the next token. Learn how earlier approaches — n-grams, RNNs, LSTMs, seq2seq — attacked that task and where they broke, and the Transformer stops being an arbitrary design and becomes an obvious fix.
next-token
the whole task
everything is prediction
RNN → attn
the arc
what broke, what fixed it
O(1)
path length
attention's key win
parallel
vs sequential
why transformers train fast
What to learn
The concepts this layer is made of. Aim for working intuition on each — you'll deepen them as you build.
1Language modeling = next-token prediction
The core framing: a language model assigns a probability to the next token given everything before it. Generation is just sampling from that distribution repeatedly. Grasp this and the whole field simplifies — training, inference, and evaluation all orbit this one objective.
2N-gram models
The simplest language models: count how often word sequences occur. Understand why they fail — they can't generalize and can't remember beyond a tiny fixed window. This motivates learned, distributed representations (embeddings).
3RNNs and LSTMs
Recurrent networks process a sequence one step at a time, carrying a hidden state. LSTMs add gates to remember longer. Learn their two fatal limits for scale: they are inherently sequential (can't parallelize over time) and struggle with long-range dependencies (the signal fades over distance).
4Seq2seq and the birth of attention
Encoder–decoder models compressed a whole input into one fixed vector — a bottleneck. Attention (Bahdanau, 2014) let the decoder look back at all input positions directly. This is the seed idea; the Transformer's insight was that attention alone — no recurrence — is enough.
5Why attention wins
Two properties: every position can reach every other in a single step (constant path length, so long-range dependencies are easy), and all positions can be computed in parallel (no time-step loop, so training saturates GPUs). These are exactly the RNN weaknesses, inverted — which is why Layer 2 (the Transformer) follows naturally.
Roadmap — the order to learn it
Read Karpathy's makemore section of Zero to Hero — you build character-level language models and feel next-token prediction in code.
Read Chris Olah's Understanding LSTMs for clear intuition on recurrence, gating, and where memory comes from.
Read The Illustrated Transformer by Jay Alammar as the bridge — it shows exactly how attention replaces recurrence.
Then go straight into Layer 2: the interactive Attention Is All You Need edition in this library.
Optional depth: the sequence-models / attention chapters of Stanford CS224N or d2l.ai.
You're ready to move on when…
- You can state the language-modeling objective in one sentence.
- You can explain the two reasons RNNs don't scale (sequential + long-range decay).
- You understand what attention does and why it fixes both problems.
- You're ready to read the Transformer architecture and have it make sense.
Resources
A curated shortlist — quality over quantity. Start at the top.