Transformer

The transformer is a neural network architecture that represents each element of a sequence in terms of every other element, using an operation called self-attention. Introduced in 2017 for machine translation, it replaced recurrence with a structure that could be trained in parallel across a whole sequence, and that change made it practical to train models of unprecedented size on unprecedented amounts of data 1.

Most frontier language models are transformers or close derivatives, and the architecture has spread well beyond text: to image recognition, speech, video generation, protein structure prediction, and the control policies of robots. It is not a model in its own right but a template — a repeating block of operations, a set of choices about how to feed data into it, and a training objective 2. That template proved portable in a way earlier sequence architectures were not.

This entry explains what the transformer replaced, how self-attention works, why the design mattered so much, what it costs, and where its limits and disputes lie.

Before 2017, the standard tools for sequence data were recurrent networks, especially the long short-term memory network. A recurrent model reads a sequence one position at a time, carrying a hidden state forward. That design respects the order of the input, but it also serialises computation: the state at position ten cannot be computed until the state at position nine exists.

Recurrence had two consequences. Training could not be parallelised across sequence positions, so long sequences and large models made training slow. And information from early positions had to survive many steps to influence later ones, which made long-range dependencies hard to learn despite gates designed to help 2.

Attention had already been added to recurrent translation models as a patch: the decoder could look back at the encoder's states instead of relying on a single fixed vector. Convolutional sequence models were more parallel but still required a path length that grew with distance. The transformer's proposal was to keep attention and discard the recurrence entirely, so that any two positions are one operation apart 1. The price of that choice is that the operation is indifferent to order, so the model has to be told where each token sits.

Each token is embedded as a vector and projected into three vectors: a query, a key, and a value. The query for one position is compared with the key of every position by a dot product, producing a score. The scores are divided by the square root of the key dimension, passed through a softmax, and used to weight the corresponding values. The result is a new representation for each position that mixes information from the positions it attended to 1.

In matrix form the operation is compact: attention is the softmax of QKᵀ divided by the scaling factor, multiplied by V. Because the same computation is applied to all positions at once, it maps efficiently onto matrix-multiply hardware. Multi-head attention repeats the operation with several independent projections and concatenates the results, letting different heads specialise in different relationships. Because the heads are computed in parallel, the extra capacity costs little in wall-clock time, which is why head count became a standard design knob.

A transformer block wraps attention in residual connections and layer normalisation, then applies a position-wise feedforward network, usually two linear layers with a nonlinearity and an expansion in between. Positional information is injected through encodings added to the input or, in most recent models, through rotations applied to queries and keys. In decoder-only models a causal mask prevents each position from attending to later ones, and training minimises the loss of predicting the next token 2.

The original transformer was an encoder-decoder for translation. The encoder read the source sentence bidirectionally; the decoder generated the target one token at a time, attending to its own past and to the encoder's output through cross-attention. This remains the shape of translation and speech-recognition systems.

BERT kept only the encoder and trained it by masking parts of the input and asking the model to reconstruct them. The resulting representations became the standard for classification, retrieval, and question answering, because a single fine-tuned model could be applied to many tasks 3.

GPT kept only the decoder and trained it to predict the next token. Scaled up, this simple objective produced a model that could follow instructions and learn from examples placed in its context, which turned the decoder-only transformer into the basis of the assistant era 4. The decoder-only design also proved the simplest to serve, because one forward pass produces one token and the cache can be reused for the next. Encoder-decoder models such as T5 remained competitive for tasks with a clear input and output, and the same block was adapted to images by splitting a picture into patches, to audio by spectrograms, and to multimodal systems by projecting other modalities into the token space 2.

The first reason is hardware fit. Self-attention and the feedforward layers are dense matrix multiplications, the operation that graphics processors and specialised accelerators perform best. Training can process many sequences and many positions in parallel, so a larger cluster converts directly into faster training rather than waiting on sequential dependencies.

The second is scaling. Empirical work found that loss falls as a power law when parameters, data, and compute are increased together, at least over the ranges studied 5. That predictability gave laboratories a reason to build bigger clusters and to trust that the investment would produce a better model. The transformer was the architecture that made those runs feasible.

The third is transfer. Pre-training on unlabelled text, followed by fine-tuning or prompting, lets one model serve many tasks, which changed the economics of machine learning. A single pre-training run could be amortised across thousands of applications, and capability improvements in the base model propagated to all of them 4. Because the same block could be trained on code, dialogue, images, or protein sequences, investment in one architecture carried across domains. The result was a shift in research and capital toward a small number of very large training runs.

Self-attention compares every position with every other, so its compute and memory grow with the square of sequence length. Doubling the context roughly quadruples the attention work in a layer. At inference, the key-value cache stores past keys and values, and its size grows with context length and batch size, so long conversations consume memory long after the prompt has been processed.

Engineering has pushed back on both costs. FlashAttention reorganises the computation to reduce movement between fast and slow GPU memory, achieving exact attention with much lower memory overhead 6. Multi-query and grouped-query attention share key-value heads across queries; sliding-window and sparse attention restrict which positions can interact; mixture-of-experts layers activate only part of the feedforward network for each token. These techniques change the constants and the memory profile, not the underlying quadratic term.

The behavioural limits are separate. A transformer predicts tokens from context; it has no built-in mechanism for verifying a claim, no persistent memory beyond its weights and prompt, and no grounded model of the world. It can be fluent and wrong at the same time. Attention weights are also not a faithful explanation of its output: different attention patterns can produce the same prediction, so reading the weights as reasons is unreliable 7. Training and serving also consume substantial electricity, which turns choices about model size and context length into choices about infrastructure.

Extending context has been one of the main engineering projects of the past few years. The original models used contexts of a few hundred tokens; current systems advertise hundreds of thousands. Getting there required positional schemes that extrapolate beyond the training range, kernels that fit larger attention matrices into memory, and serving systems that page the key-value cache like virtual memory.

Longer context is not the same as better use of context. Models attend unevenly across a long input, and information placed in the middle of a very long prompt is often used less reliably than information at the start or end. Retrieval, summarisation, and structured prompts remain necessary even when the window is large enough to hold the document.

Alternative architectures aim at the quadratic term. State-space models such as Mamba process sequences with linear scaling and a recurrent state, and hybrid designs mix a few attention layers with many state-space layers 8. They are competitive at some sizes and workloads, particularly for long sequences, but no alternative has yet displaced attention at the frontier. Whether that reflects a fundamental advantage of attention or the enormous investment already sunk into it is an open question.

The first dispute is about explanation. Attention is often described as showing what a model looks at, and the weights are visualised as if they were reasons. Careful experiments found that attention distributions can be altered substantially without changing predictions, and that they do not reliably identify the input features responsible for an output 7. Interpretability has since moved toward analysing circuits, features, and the residual stream rather than reading attention maps.

The second dispute is about causation. One account credits the architecture for the capability boom: parallel attention made scale practical, and scale produced the gains. Another credits data, compute, and training objectives, arguing that the architecture is a detail that happened to be available when the budget arrived. The scaling laws are consistent with both, since they describe the returns to scale without isolating the architectural contribution 5.

The third is about trajectory. Critics point to the quadratic cost, the lack of persistent memory, and brittle multi-step reasoning, and argue that the design is nearing its limits. Proponents point to continued gains from longer context, better post-training, and tool use, and note that every predicted plateau so far has been postponed. The evidence does not yet settle the question. What is clear is that headline capability claims are usually vendor-reported, that benchmark questions leak into training data, and that comparisons at equal compute and equal data are rare.

The transformer's influence is visible in vocabulary as much as in products. Prefill and decode, key-value cache, context window, and attention head are terms that describe the internal economics of serving a transformer, and they now circulate as if they described computing in general.

The architecture also became the substrate for other sciences. Protein structure models use attention over amino-acid residues; image generators use transformer blocks in place of convolutional layers; speech systems pair an encoder with a decoder; robotics models turn camera images and instructions into actions through vision-language transformers. The block proved general because it makes few assumptions about the structure of its input beyond a sequence of vectors 2.

Whether the transformer is the final architecture is a different question from whether it was the decisive one. It made scale practical, made transfer routine, and gave a generation of systems a common shape. Those are historical achievements, and they are separable from the engineering limits that will eventually produce a successor. What replaces it will probably keep the parts that worked: learned representations, massive pre-training, and a hardware-friendly inner loop.