31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USAarXiv:1706.03762 ↗

Attention Is All
You Need.

Ashish Vaswani (Google Brain) · Noam Shazeer (Google Brain) · Niki Parmar (Google Research) · Jakob Uszkoreit (Google Research) · Llion Jones (Google Research) · Aidan N. Gomez (University of Toronto) · Łukasz Kaiser (Google Brain) · Illia Polosukhin

Equal contribution — the listing order is random. See the paper's footnote for each author's individual contributions.

This paper introduced the Transformer, the first sequence transduction model built entirely on attention — no recurrence, no convolutions. By letting every position attend directly to every other position, it trains dramatically faster through parallelization while setting new state-of-the-art BLEU scores on WMT 2014 English–German (28.4) and English–French (41.8) machine translation, at a fraction of the training cost of previous models. The architecture it describes became the foundation of BERT, GPT, and essentially all modern large language models.

Read the PDF tensor2tensor on GitHubReading level:

28.4

BLEU · EN→DE

+2.0 over best ensemble

41.8

BLEU · EN→FR

single-model SOTA

3.5 d

training time (big)

8× P100 GPUs

<¼

training cost

of previous SOTA

Introduction

Recurrent models (LSTMs, GRUs) were state of the art for sequence transduction, but their computation is inherently sequential: the hidden state $h_{t}$ depends on $h_{t - 1}$ , which prevents parallelization within a training example and becomes critical at longer sequence lengths. Tricks like factorization and conditional computation improved efficiency but did not remove the fundamental constraint.

mechanisms had already proven valuable — but almost always bolted onto an RNN. The Transformer removes the recurrence entirely and relies solely on attention to draw global dependencies between input and output, reaching a new state of the art in translation quality after as little as twelve hours on eight P100 GPUs.

Background

The Extended Neural GPU, ByteNet and ConvS2S all reduce sequential computation by using convolutions to compute representations in parallel — but the number of operations needed to relate two arbitrary positions grows with their distance: linearly for ConvS2S and logarithmically for ByteNet, making distant dependencies harder to learn.

The Transformer reduces this to a constant number of operations. The cost is reduced effective resolution from averaging attention-weighted positions — counteracted with . itself was not new (it had been used in reading comprehension, summarization, and entailment), but the Transformer is the first transduction model relying entirely on self-attention without sequence-aligned RNNs or convolution.

Model Architecture

The Transformer follows the classic structure: the encoder maps an input sequence $(x_{1}, \dots, x_{n})$ to continuous representations $z$ , and the decoder generates the output $(y_{1}, \dots, y_{m})$ , consuming previously generated symbols when producing the next.

Both stacks use $N = 6$ layers with and around each sub-layer: $LayerNorm (x + Sublayer (x))$ , with all sub-layers outputting dimension $d_{model} = 512$ . Encoder layers pair self-attention with a position-wise feed-forward network; decoder layers add a third sub-layer that attends over the encoder output, and mask their self-attention so position $i$ can only see positions $< i$ .

Attention is computed as scaled dot-product attention over , run $h = 8$ times in parallel () over learned projections. Because the model contains no recurrence or convolution, sinusoidal positional encodings are added to the embeddings to inject order information.

3.1 · The encoder–decoder stacks

Figure 1 — The Transformer architecture

Multi-Head Self-Attention

Encoder self-attention: queries, keys and values all come from the previous encoder layer, so every position attends to every position in the input — in a single O(1)-path step. 8 heads, each 64-dimensional.

Click any block — or press Trace the data flow — to follow a sentence through the model.

AttentionFeed-forwardEmbeddingsOutput head

Redrawn from Figure 1 of the paper. Left: encoder stack (N = 6 layers, each with self-attention + feed-forward). Right: decoder stack (N = 6 layers with masked self-attention, cross-attention over the encoder output, and feed-forward). Every sub-layer is wrapped in a residual connection and layer normalization (curved side arrows).

3.2 · Attention

An attention function maps a query and a set of key–value pairs to an output: a weighted sum of the values, where each weight comes from a compatibility score between the query and the corresponding key. The Transformer's flavor is scaled dot-product attention:

Attention (Q, K, V) = softmax (\frac{Q K ^{⊤}}{d _{k}}) V

(1)

Scaled Dot-Product Attention

Multi-Head Attention

Rather than one attention over $d_{model}$ -dimensional vectors, the model projects queries, keys and values $h = 8$ times into 64-dimensional subspaces and attends in parallel — same total cost, richer behavior:

MultiHead (Q, K, V) head_{i} = Concat (head_{1}, \dots, head_{h}) W^{O} = Attention (Q W_{i}^{Q}, K W_{i}^{K}, V W_{i}^{V})

Try it — what does “it” refer to?

Hover or tap the highlighted words to see illustrative self-attention weights (blue depth = attention received). When the model processes “it”, attention flows strongly to “animal” — resolving the reference in a single step, no matter the distance.

Weights are illustrative, modeled on the coreference visualizations in the paper's appendix. In the real model, 8 heads per layer each learn different patterns like this.

3.3 · Position-wise feed-forward networks

FFN (x) = max (0, x W_{1} + b_{1}) W_{2} + b_{2}

(2)

Applied to each position separately and identically — two linear maps with a ReLU between them, expanding from $d_{model} = 512$ to $d_{f f} = 2048$ and back. Parameters differ from layer to layer.

3.4 · Embeddings and softmax

Input embedding, output embedding, and the pre-softmax linear projection share one weight matrix; inside the embedding layers the weights are multiplied by $d_{model}$ .

3.5 · Positional encoding

P E_{(p os, 2 i)} P E_{(p os, 2 i + 1)} = sin (p os /1000 0^{2 i / d_{model}}) = cos (p os /1000 0^{2 i / d_{model}})

(3)

The positional encoding, visualized

Each row is a position (0–47); each column one of the first 40 embedding dimensions. Low dimensions oscillate fast; high dimensions slowly — together they form a unique, smoothly-varying fingerprint for every position. Hover any cell.

PE(pos, 2i) = sin(pos / 10000^(2i/512))

−1 0 +1

Why Self-Attention

The paper compares self-attention to recurrent and convolutional layers on three criteria: total computational complexity per layer, minimum number of sequential operations (a proxy for parallelizability), and maximum path length between any two positions — since shorter paths make long-range dependencies easier to learn.

Self-attention wins on paths and parallelism: $O (1)$ sequential operations and $O (1)$ path length, versus $O (n)$ for recurrence. Per-layer cost is $O (n^{2} \cdot d)$ — faster than a recurrent layer's $O (n \cdot d^{2})$ whenever sequence length $n$ is smaller than representation dimension $d$ , which holds for typical sentence representations. A restricted variant (attend within a window of size $r$ ) trades path length $O (n / r)$ for scalability to very long sequences.


Self-Attention	$O (n^{2} \cdot d)$	$O (1)$	$O (1)$
Recurrent	$O (n \cdot d^{2})$	$O (n)$	$O (n)$
Convolutional	$O (k \cdot n \cdot d^{2})$	$O (1)$	$O (lo g_{k} (n))$
Self-Attention (restricted)	$O (r \cdot n \cdot d)$	$O (1)$	$O (n / r)$

Table 1 — Maximum path lengths, per-layer complexity and minimum number of sequential operations for different layer types. n = sequence length, d = representation dimension, k = kernel size, r = neighborhood size.

Before / after — drag to compare information flow

The core reason attention wins: the path between distant words collapses from O(n) hops to a single hop.

⇄

BEFORE · RNNAFTER · TRANSFORMER

Training

Data: WMT 2014 English–German (≈4.5M sentence pairs, with a shared ~37,000-token vocabulary) and WMT 2014 English–French (36M sentences, 32,000 vocabulary). Batches contained ≈25,000 source + 25,000 target tokens, grouped by approximate sequence length.

Optimization: ( $β_{1} = 0.9$ , $β_{2} = 0.98$ , $ϵ = 1 0^{- 9}$ ) with a learning rate that increases linearly for 4,000 steps and then decays proportionally to the inverse square root of the step number. Regularization: $P_{d r o p} = 0.1$ on every sub-layer output and on the embedding sums, plus $ϵ_{l s} = 0.1$ — which hurts perplexity but improves accuracy and BLEU.

The training pipeline, step by step

Step 0
Tokenize & batch
Sentences are encoded with byte-pair encoding (37K shared tokens EN-DE / 32K word-pieces EN-FR) and batched by approximate length — ~25,000 source + 25,000 target tokens per batch.
Steps 0 → 4,000
Learning-rate warmup
The learning rate rises linearly for the first 4,000 steps — stabilizing early training — then decays as the inverse square root of the step number.
Steps 4,000 → 100K/300K
Main training
Adam (β₁ 0.9, β₂ 0.98, ε 10⁻⁹) with residual dropout 0.1 and label smoothing 0.1 on 8× P100 GPUs. Base: 0.4 s/step for 100K steps (~12 h). Big: 1.0 s/step for 300K steps (~3.5 days).
Final hours
Checkpoint averaging
The final model is the average of the last 5 checkpoints for base (written every 10 minutes) or the last 20 for big — a cheap ensemble effect.
Inference
Beam-search decoding
Beam size 4 with length penalty α = 0.6; maximum output length input + 50, terminating early when possible. Result: 28.4 BLEU EN→DE, 41.8 BLEU EN→FR.

The learning-rate schedule

l r a t e = d_{model}^{- 0.5} \cdot min (s t e p_n u m^{- 0.5}, s t e p_n u m \cdot w a r m u p_s t e p s^{- 1.5})

(4)

The learning-rate schedule

lrate = d_model⁻⁰·⁵ · min(step⁻⁰·⁵, step · 4000⁻¹·⁵) — linear warmup, then inverse-square-root decay

Hover the curve. The warmup phase (first 4,000 steps) protects the post-norm residual stack while activations settle; this schedule became famous as the “Noam schedule”.

Hyperparameters at a glance

N

Layers per stack

base

6

big

6

Identical layers in both the encoder and the decoder stack.

d_{model}

Model dimension

base

512

big

1024

Width of every sub-layer output and embedding.

d_{f f}

Feed-forward inner size

base

2048

big

4096

Hidden width of the position-wise FFN (4× d_model).

h

Attention heads

base

8

big

16

Parallel attention heads per attention sub-layer.

d_{k} = d_{v}

Per-head dimension

base

64

big

64

d_model / h — keeps multi-head cost equal to one full-width head.

P_{d r o p}

Dropout

base

0.1

big

0.3

On each sub-layer output and on embedding + positional-encoding sums. (Big EN-FR used 0.1.)

ϵ_{l s}

Label smoothing

base

0.1

big

0.1

Hurts perplexity, improves accuracy and BLEU.

steps

Training steps

base

100K (~12 h)

big

300K (~3.5 d)

On 8× NVIDIA P100 GPUs; 0.4 s vs 1.0 s per step.

β_{1}, β_{2}, ϵ

Adam parameters

base

0.9 / 0.98 / 10⁻⁹

big

0.9 / 0.98 / 10⁻⁹

With warmup_steps = 4000 in the lrate schedule.

batch

Batch size

base

≈25K + 25K tokens

big

≈25K + 25K tokens

Source + target tokens, batched by approximate sequence length.

beam

Beam search

base

4, α = 0.6

big

4, α = 0.6

Max output length = input + 50, with early termination.

params

Parameters

base

65M

big

213M

Total trainable parameters.

Datasets

WMT 2014 English–German

Machine translation

≈4.5M sentence pairs · ~37,000 shared BPE tokens

The standard WMT 2014 EN-DE benchmark. Sentences are encoded with byte-pair encoding over a source–target shared vocabulary, letting the model handle rare words compositionally. Evaluated on newstest2014 (test) and newstest2013 (dev).

WMT 2014 English–French

Machine translation

36M sentences · 32,000 word-piece vocabulary

The significantly larger WMT 2014 EN-FR dataset. Tokens are split into word-pieces. The big Transformer trained here used dropout 0.1 instead of 0.3 and reached 41.8 BLEU — single-model state of the art.

Penn Treebank (WSJ)

English constituency parsing

≈40K training sentences · 16K tokens

Wall Street Journal portion of the Penn Treebank, used to test whether the Transformer generalizes beyond translation. A 4-layer, d_model = 1024 Transformer reached 91.3 F1 training on WSJ alone.

Semi-supervised parsing corpora

English constituency parsing

≈17M sentences · 32K tokens

The high-confidence and BerkeleyParser corpora, used for the semi-supervised parsing setting — lifting the Transformer to 92.7 F1, above all previously reported models except the generative RNNG.

Results

Machine translation (Table 2): the big Transformer beats the best previously reported models — including ensembles — on WMT 2014 EN→DE by more than 2.0 BLEU (28.4), and reaches 41.8 on EN→FR at less than 1/4 the training cost of the previous SOTA. Even the base model surpasses all previously published single models and ensembles on EN→DE at a fraction of the training cost. Inference used (beam 4, length penalty $α = 0.6$ ) and checkpoint averaging.

Ablations (Table 3): single-head attention is 0.9 BLEU worse than the best setting; too many heads also degrades quality. Reducing $d_{k}$ hurts (compatibility scoring is not trivial); bigger models and dropout both help decisively; learned positional embeddings are indistinguishable from sinusoids.

Generalization (Table 4): a 4-layer Transformer ( $d_{model} = 1024$ ) applied to English constituency — despite task-specific challenges (strong structural constraints, outputs longer than inputs) — outperforms all previously reported models except one, even when trained on only the 40k-sentence WSJ set, and reaches 92.7 F1 semi-supervised.

6.1 · Machine translation

BLEU on WMT 2014 English → German (newstest2014)

Higher is better · blue = the Transformer, gray = prior single models and ensembles

The big Transformer beats even the best ensembles by more than 2 BLEU — and the base model already outperforms every previously published model.


ByteNet	23.75	—	—	—
Deep-Att + PosUnk	—	39.2	—	1.0 · 10^20
GNMT + RL	24.6	39.92	2.3 · 10^19	1.4 · 10^20
ConvS2S	25.16	40.46	9.6 · 10^18	1.5 · 10^20
MoE	26.03	40.56	2.0 · 10^19	1.2 · 10^20
Deep-Att + PosUnk Ensembleensemble	—	40.4	—	8.0 · 10^20
GNMT + RL Ensembleensemble	26.3	41.16	1.8 · 10^20	1.1 · 10^21
ConvS2S Ensembleensemble	26.36	41.29	7.7 · 10^19	1.2 · 10^21
Transformer (base)	27.3	38.1	3.3 · 10^18	3.3 · 10^18
Transformer (big)	28.4	41.8	2.3 · 10^19	2.3 · 10^19

Table 2 — The Transformer achieves better BLEU scores than previous state-of-the-art models on the WMT 2014 English→German and English→French newstest2014 tests, at a fraction of the training cost (FLOPs estimated as training time × #GPUs × sustained single-precision capacity).

6.2 · Model variations (ablations)

Ablation (A): number of attention heads

Dev-set BLEU (newstest2013) at constant compute — d_k = d_v = d_model / h

One head is 0.9 BLEU worse than the best setting; quality also drops at 32 heads. The paper's pick of h = 8 sits at the sweet spot.


basebase	8	512	2048	0.1	4.92	25.8	65
Ah = 1	1	512	2048	0.1	5.29	24.9	—
Ah = 4	4	512	2048	0.1	5	25.5	—
Ah = 16	16	512	2048	0.1	4.91	25.8	—
Ah = 32	32	512	2048	0.1	5.01	25.4	—
Bdk = 16	8	512	2048	0.1	5.16	25.1	58
Bdk = 32	8	512	2048	0.1	5.01	25.4	60
CN = 2	8	512	2048	0.1	6.11	23.7	36
CN = 4	8	512	2048	0.1	5.19	25.3	50
CN = 8	8	512	2048	0.1	4.88	25.5	80
Cd_model = 256	8	256	2048	0.1	5.75	24.5	28
Cd_model = 1024	8	1024	2048	0.1	4.66	26	168
Cd_ff = 1024	8	512	1024	0.1	5.12	25.4	53
Cd_ff = 4096	8	512	4096	0.1	4.75	26.2	90
DP_drop = 0.0	8	512	2048	0	5.77	24.6	—
DP_drop = 0.2	8	512	2048	0.2	4.95	25.5	—
Dε_ls = 0.0	8	512	2048	0.1	4.67	25.3	—
Dε_ls = 0.2	8	512	2048	0.1	5.47	25.7	—
Elearned positions	8	512	2048	0.1	4.92	25.7	—
bigbig	16	1024	4096	0.3	4.33	26.4	213

Table 3 — Variations on the Transformer architecture. Unlisted values are identical to the base model. All metrics are on the English→German translation development set, newstest2013 (perplexities are per-wordpiece).

6.3 · English constituency parsing


Vinyals & Kaiser et al. (2014)	WSJ only, discriminative	88.3
Petrov et al. (2006)	WSJ only, discriminative	90.4
Zhu et al. (2013)	WSJ only, discriminative	90.4
Dyer et al. (2016)	WSJ only, discriminative	91.7
Transformer (4 layers)	WSJ only, discriminative	91.3
Zhu et al. (2013)	semi-supervised	91.3
Huang & Harper (2009)	semi-supervised	91.3
McClosky et al. (2006)	semi-supervised	92.1
Vinyals & Kaiser et al. (2014)	semi-supervised	92.1
Transformer (4 layers)	semi-supervised	92.7
Luong et al. (2015)	multi-task	93
Dyer et al. (2016)	generative	93.3

Table 4 — The Transformer generalizes well to English constituency parsing (results on Section 23 of the WSJ Penn Treebank).

Conclusion

The Transformer is presented as the first sequence transduction model based entirely on attention, replacing recurrent layers in encoder–decoder architectures with multi-head self-attention. It trains significantly faster than recurrent or convolutional architectures and set new SOTA on both WMT 2014 tasks — on EN→DE beating even all previously reported ensembles.

Planned future work: extending to input/output modalities beyond text (images, audio, video), local, restricted attention for large inputs, and making generation less sequential. The code was released in the tensor2tensor library.

Code & Resources

tensorflow/tensor2tensor→

The official repository referenced in the paper — the code used to train and evaluate all models, released to accelerate research. (Its modern successor is Trax / T5X.)

📄 arXiv:1706.03762→

The original paper — abstract, full PDF, and revision history. This site is a faithful interactive companion, not a substitute: every number here comes from the paper.

Equation (1), as runnable PyTorch

The paper contains no code listings, so here is a minimal, faithful implementation of scaled dot-product and multi-head attention exactly as defined in §3.2 — a good starting point for reimplementing the paper.

import torch, math

def scaled_dot_product_attention(Q, K, V, mask=None):
    """Attention(Q, K, V) = softmax(QK^T / sqrt(d_k)) V   — Eq. (1)"""
    d_k = Q.size(-1)
    scores = Q @ K.transpose(-2, -1) / math.sqrt(d_k)
    if mask is not None:                      # decoder: block future positions
        scores = scores.masked_fill(mask == 0, float("-inf"))
    weights = torch.softmax(scores, dim=-1)   # attention distribution
    return weights @ V, weights

class MultiHeadAttention(torch.nn.Module):
    def __init__(self, d_model=512, h=8):
        super().__init__()
        assert d_model % h == 0
        self.d_k, self.h = d_model // h, h    # d_k = d_v = d_model / h = 64
        self.W_q = torch.nn.Linear(d_model, d_model)
        self.W_k = torch.nn.Linear(d_model, d_model)
        self.W_v = torch.nn.Linear(d_model, d_model)
        self.W_o = torch.nn.Linear(d_model, d_model)

    def forward(self, query, key, value, mask=None):
        B = query.size(0)
        # project, then split into h heads of size d_k
        Q, K, V = (W(x).view(B, -1, self.h, self.d_k).transpose(1, 2)
                   for W, x in ((self.W_q, query), (self.W_k, key), (self.W_v, value)))
        out, _ = scaled_dot_product_attention(Q, K, V, mask)
        out = out.transpose(1, 2).contiguous().view(B, -1, self.h * self.d_k)
        return self.W_o(out)                  # Concat(head_1..head_h) W^O

Glossary

Every dotted-underlined term on this page links back here. Hover any of them in context, or browse the full definitions below.

Attention

A function mapping a query and a set of key–value pairs to an output, where the output is a weighted sum of the values and each weight is computed by a compatibility function between the query and the corresponding key. It lets a model dynamically decide which parts of a sequence to draw information from.

Self-attention

Also called intra-attention: an attention mechanism where queries, keys and values all come from the same sequence, relating different positions to compute a representation of that sequence. The Transformer is built entirely from self-attention (plus cross-attention in the decoder).

Queries, Keys, Values

Each token embedding is linearly projected into a query (what this position is looking for), a key (what this position offers for matching), and a value (the information passed along once matched). Attention weights come from query–key dot products; the output is the weight-averaged values.

Multi-head attention

Instead of a single attention over d_model-dimensional vectors, the model projects Q, K, V h times into lower-dimensional subspaces (d_k = d_model/h), runs attention in parallel in each, concatenates the outputs and projects once more. This lets the model jointly attend to different representation subspaces at different positions — single-head averaging inhibits this.

Positional encoding

Because attention is permutation-invariant, the Transformer adds a deterministic sinusoidal pattern to each embedding: PE(pos,2i) = sin(pos/10000^(2i/d_model)), PE(pos,2i+1) = cos(...). Wavelengths form a geometric progression, and for any offset k, PE(pos+k) is a linear function of PE(pos) — making relative positions easy to compute.

Encoder–decoder

The dominant structure for sequence transduction: an encoder maps an input sequence (x1..xn) to continuous representations z, and a decoder generates an output sequence (y1..ym) one element at a time, conditioning on z and on previously generated symbols.

Auto-regressive

A generation regime where the model produces the output sequence one token at a time, consuming its own previously generated tokens as additional input for the next prediction. The Transformer decoder is auto-regressive at inference; masking preserves this property during parallel training.

Residual connection

A skip connection (He et al., 2016) around each sub-layer: the output becomes LayerNorm(x + Sublayer(x)). Residuals let gradients flow directly through deep stacks and are one reason 6-layer (and much deeper) Transformers train stably.

Layer normalization

Layer normalization (Ba et al., 2016) rescales each position's activation vector to have stable statistics, applied after every residual addition in the Transformer. (Later work often moves it before the sub-layer — 'pre-norm'.)

Softmax

The exponential-normalization function converting attention scores into weights, and decoder outputs into next-token probabilities. Large-magnitude inputs push softmax into saturated regions with vanishing gradients — the reason attention scores are scaled by 1/√d_k.

Position-wise FFN

FFN(x) = max(0, xW1 + b1)W2 + b2, applied to each position separately with the same weights within a layer (equivalent to two 1×1 convolutions). Inner dimension d_ff = 2048 for the base model — 4× d_model.

BLEU

The standard automatic metric for machine translation, measuring n-gram precision of the system output against human reference translations, with a brevity penalty. The paper's headline numbers: 28.4 BLEU on WMT 2014 EN→DE and 41.8 on EN→FR.

Perplexity

A language-modeling metric: the exponential of the per-token cross-entropy. Table 3 reports per-wordpiece perplexities on the dev set. Note label smoothing worsens perplexity while improving BLEU — the two metrics can disagree.

Byte-pair encoding (BPE)

A data-driven tokenization (Sennrich et al., 2016) that splits rare words into frequent sub-word units, keeping vocabulary small (≈37,000 shared tokens for EN-DE) while avoiding out-of-vocabulary failures.

Word-piece

A sub-word segmentation method (used in GNMT, Wu et al. 2016) that chooses merges to maximize language-model likelihood. The EN-FR models use a 32,000 word-piece vocabulary.

Label smoothing

A regularizer (Szegedy et al., 2016) that replaces the one-hot target with a mixture: 1−ε on the true token and ε spread across the vocabulary. It makes the model less over-confident: perplexity worsens, but accuracy and BLEU improve.

Beam search

An approximate search for the highest-probability output sequence: at each step, keep the top-k (beam size) partial hypotheses. The paper uses beam 4 with length penalty α = 0.6, maximum output length input + 50, and early termination.

Dropout

Applied to each sub-layer's output before the residual add-and-norm, and to the sums of embeddings and positional encodings. Table 3 row (D) shows it is essential: removing it costs 1.2 BLEU on the dev set.

Adam

The Adam optimizer (Kingma & Ba, 2015) with unusually high β₂ = 0.98, paired with the warmup + inverse-square-root learning-rate schedule that later became known as the 'Noam schedule'.

Warmup

The schedule lrate = d_model^-0.5 · min(step^-0.5, step·warmup^-1.5) increases the learning rate linearly for warmup_steps = 4000, then decays it proportionally to the inverse square root of the step number — stabilizing early training with post-norm residuals.

FLOPs

The paper estimates training cost as training time × number of GPUs × each GPU's sustained single-precision floating-point capacity. The base Transformer used ~3.3×10¹⁸ FLOPs — orders of magnitude less than previous SOTA models.

Constituency parsing

The task of producing a sentence's phrase-structure tree (as bracketed constituents). Outputs are subject to strong structural constraints and are longer than the input — a challenging generalization test that the Transformer passes with 91.3/92.7 F1.

LSTM

A gated recurrent architecture (Hochreiter & Schmidhuber, 1997) designed to preserve information across long sequences. LSTMs and GRUs were the state of the art for sequence transduction that the Transformer displaced.

Ablation

The systematic study in Table 3: vary attention heads, key size, model size, dropout, label smoothing, and positional encoding one factor at a time, measuring the effect on dev-set perplexity and BLEU.

d_model

The uniform representation width flowing through the network. All sub-layers and embeddings output d_model dimensions to make residual connections possible.

FAQ

Previous state-of-the-art models used attention only as a helper on top of recurrent or convolutional networks. The title is the thesis: keep the attention, delete everything else. The Transformer contains no recurrence and no convolution — attention (plus feed-forward layers, residuals and positional encodings) is sufficient to beat all previous models.

If query and key components are independent with mean 0 and variance 1, their dot product has mean 0 but variance d_k — so scores grow with dimension. Large scores push the softmax into saturated regions where gradients vanish. Dividing by √d_k restores unit variance and keeps gradients healthy. Table 3 row (B) shows attention quality is sensitive to key size, supporting this reasoning.

A single attention distribution averages everything a position needs into one mixture, which inhibits attending to multiple distinct relationships at once. Eight heads in 64-dimensional subspaces (same total cost) let the model attend simultaneously to different representation subspaces — e.g., one head tracking syntax, another coreference. Empirically, one head costs 0.9 BLEU; 32 heads also hurts.

Two mechanisms. During training the decoder's self-attention is masked: connections to later positions are set to −∞ before the softmax, so position i receives exactly zero weight from positions > i. And the target sequence fed to the decoder is shifted right by one position, so predicting token i only ever conditions on tokens 1..i−1.

Both work equally well (Table 3, row E: 25.7 vs 25.8 BLEU). The authors chose sinusoids for two properties: for any offset k, PE(pos+k) is a linear function of PE(pos), making relative-position attention easy to learn; and fixed sinusoids may extrapolate to sequence lengths longer than any seen during training.

No — it's a trade. Self-attention costs O(n²·d) per layer versus O(n·d²) for recurrence, so it's cheaper only when sequence length n is smaller than model dimension d (true for typical sentences: n ≈ 20–70, d = 512). For very long sequences the paper proposes restricted attention over a window of size r, at the cost of growing the maximum path length to O(n/r) — foreshadowing today's sparse and sliding-window attention.

Base: d_model = 512, d_ff = 2048, 8 heads, dropout 0.1, 65M parameters, 100K steps (~12 hours on 8 P100s), 27.3 BLEU EN→DE. Big: d_model = 1024, d_ff = 4096, 16 heads, dropout 0.3, 213M parameters, 300K steps (~3.5 days), 28.4 BLEU. Both keep N = 6 layers and d_k = d_v = 64 per head.

Label smoothing trains the model against softened targets, so it deliberately never assigns full probability to the correct token — that mathematically worsens perplexity. But the resulting less-over-confident distributions produce better-calibrated scores for beam search, yielding higher translation accuracy and BLEU. It's a reminder that the training objective and the task metric are not the same thing.

It became arguably the most influential neural architecture ever. The encoder stack (with masked-token pretraining) became BERT; the decoder stack (scaled up) became GPT; encoder–decoder variants became T5. The paper's own future-work list — other modalities, restricted attention, less-sequential generation — became Vision Transformers, sparse attention, and parallel decoding research.

References

The works the paper builds on — every entry links to the cited paper.

[1]
Ba, J. L., Kiros, J. R., & Hinton, G. E. (2016). Layer normalization. arXiv:1607.06450
[2]
Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. ICLR 2015 · arXiv:1409.0473
[3]
Cheng, J., Dong, L., & Lapata, M. (2016). Long short-term memory-networks for machine reading. EMNLP 2016 · arXiv:1601.06733
[4]
Cho, K., van Merrienboer, B., Gulcehre, C., et al. (2014). Learning phrase representations using RNN encoder–decoder for statistical machine translation. EMNLP 2014 · arXiv:1406.1078
[5]
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014). Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv:1412.3555
[6]
Dyer, C., Kuncoro, A., Ballesteros, M., & Smith, N. A. (2016). Recurrent neural network grammars. NAACL 2016 · arXiv:1602.07776
[7]
Gehring, J., Auli, M., Grangier, D., Yarats, D., & Dauphin, Y. N. (2017). Convolutional sequence to sequence learning (ConvS2S). ICML 2017 · arXiv:1705.03122
[8]
He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. CVPR 2016 · arXiv:1512.03385
[9]
Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation 9(8)
[10]
Huang, Z., & Harper, M. (2009). Self-training PCFG grammars with latent annotations across languages. EMNLP 2009
[11]
Kaiser, Ł., & Sutskever, I. (2016). Neural GPUs learn algorithms. ICLR 2016 · arXiv:1511.08228
[12]
Kalchbrenner, N., Espeholt, L., Simonyan, K., et al. (2016). Neural machine translation in linear time (ByteNet). arXiv:1610.10099
[13]
Kingma, D. P., & Ba, J. (2015). Adam: A method for stochastic optimization. ICLR 2015 · arXiv:1412.6980
[14]
Kuchaiev, O., & Ginsburg, B. (2017). Factorization tricks for LSTM networks. ICLR Workshop 2017 · arXiv:1703.10722
[15]
Lin, Z., Feng, M., dos Santos, C. N., et al. (2017). A structured self-attentive sentence embedding. ICLR 2017 · arXiv:1703.03130
[16]
Luong, M.-T., Le, Q. V., Sutskever, I., Vinyals, O., & Kaiser, Ł. (2016). Multi-task sequence to sequence learning. ICLR 2016 · arXiv:1511.06114
[17]
Luong, M.-T., Pham, H., & Manning, C. D. (2015). Effective approaches to attention-based neural machine translation. EMNLP 2015 · arXiv:1508.04025
[18]
McClosky, D., Charniak, E., & Johnson, M. (2006). Effective self-training for parsing. NAACL 2006
[19]
Parikh, A., Täckström, O., Das, D., & Uszkoreit, J. (2016). A decomposable attention model for natural language inference. EMNLP 2016 · arXiv:1606.01933
[20]
Paulus, R., Xiong, C., & Socher, R. (2017). A deep reinforced model for abstractive summarization. arXiv:1705.04304
[21]
Petrov, S., Barrett, L., Thibaux, R., & Klein, D. (2006). Learning accurate, compact, and interpretable tree annotation. ACL 2006
[22]
Press, O., & Wolf, L. (2017). Using the output embedding to improve language models. EACL 2017 · arXiv:1608.05859
[23]
Sennrich, R., Haddow, B., & Birch, A. (2016). Neural machine translation of rare words with subword units (BPE). ACL 2016 · arXiv:1508.07909
[24]
Shazeer, N., Mirhoseini, A., Maziarz, K., et al. (2017). Outrageously large neural networks: the sparsely-gated mixture-of-experts layer. ICLR 2017 · arXiv:1701.06538
[25]
Srivastava, N., Hinton, G. E., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: a simple way to prevent neural networks from overfitting. JMLR 15(1)
[26]
Sukhbaatar, S., Szlam, A., Weston, J., & Fergus, R. (2015). End-to-end memory networks. NeurIPS 2015 · arXiv:1503.08895
[27]
Sutskever, I., Vinyals, O., & Le, Q. V. (2014). Sequence to sequence learning with neural networks. NeurIPS 2014 · arXiv:1409.3215
[28]
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016). Rethinking the Inception architecture for computer vision (label smoothing). CVPR 2016 · arXiv:1512.00567
[29]
Vinyals, O., Kaiser, Ł., Koo, T., Petrov, S., Sutskever, I., & Hinton, G. (2015). Grammar as a foreign language. NeurIPS 2015 · arXiv:1412.7449
[30]
Wu, Y., Schuster, M., Chen, Z., et al. (2016). Google's neural machine translation system: bridging the gap between human and machine translation (GNMT). arXiv:1609.08144
[31]
Zhou, J., Cao, Y., Wang, X., Li, P., & Xu, W. (2016). Deep recurrent models with fast-forward connections for neural machine translation (Deep-Att). TACL 2016 · arXiv:1606.04199
[32]
Zhu, M., Zhang, Y., Chen, W., Zhang, M., & Zhu, J. (2013). Fast and accurate shift-reduce constituent parsing. ACL 2013

Cite This Paper

@inproceedings{vaswani2017attention,
  title     = {Attention Is All You Need},
  author    = {Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and
               Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N. and
               Kaiser, {\L}ukasz and Polosukhin, Illia},
  booktitle = {Advances in Neural Information Processing Systems 30 (NIPS 2017)},
  pages     = {5998--6008},
  year      = {2017},
  url       = {https://arxiv.org/abs/1706.03762}
}

Attention Is AllYou Need.

Introduction

Background

Model Architecture

3.1 · The encoder–decoder stacks

3.2 · Attention

3.3 · Position-wise feed-forward networks

3.4 · Embeddings and softmax

3.5 · Positional encoding

Why Self-Attention

Training

The training pipeline, step by step

The learning-rate schedule

Hyperparameters at a glance

Datasets

Results

6.1 · Machine translation

6.2 · Model variations (ablations)

6.3 · English constituency parsing

Conclusion

Code & Resources

Glossary

FAQ

References

Cite This Paper

Attention Is All
You Need.