Large language model

A large language model (LLM) is a neural network trained to predict the next token in a sequence of text, and scaled until that simple objective produces useful language behaviour. The models are large in two senses: they contain billions of parameters, and they are trained on corpora measured in trillions of tokens.

Trained this way, a language model can draft, summarise, translate, answer questions, and write code. It can also state falsehoods fluently and has no reliable sense of what it does not know. Both the competence and the failures follow from the same statistical objective.

This entry covers the architecture, the data and training pipeline, demonstrated capabilities, reasoning and tool use, persistent limitations, the economics of scale and access, and the politics of evaluation.

Most current language models use the transformer architecture, introduced in 2017 1. The dominant variant is decoder-only: a stack of identical blocks, each combining masked self-attention with a feedforward network, wrapped in residual connections and normalisation, trained to predict the next token from the preceding ones.

Text is split into tokens — subword units drawn from a fixed vocabulary of roughly 50,000 to 200,000 entries — and each token is mapped to a vector. Self-attention lets every position weigh every earlier position, so the model can use distant context, while positional encodings, usually rotary, preserve order. The feedforward layer, which holds most of the parameters, transforms each position independently.

The output layer produces a probability distribution over the vocabulary, and generation samples from it, often with a temperature or top-p truncation that trades diversity for reliability. Training compares that distribution with the actual next token and penalises the difference.

Parameter counts range from a few hundred million to over a trillion. Many recent models use mixture-of-experts layers, activating only part of the network for each token, which cuts computation per token at the cost of memory. Attention over long contexts is expensive because its cost grows with the square of sequence length, so serving systems cache intermediate states and increasingly use sparse or linear attention variants.

Pretraining corpora are assembled from web crawls, books, academic papers, source code, reference works, and licensed archives. The mixture matters: code improves reasoning about formal systems, reference text improves factual recall, and duplicated or low-quality pages degrade both. Pipelines filter for language, remove boilerplate and duplicates, and increasingly classify quality with a model.

The GPT-3 paper described a model trained on roughly 300 billion tokens 2; later systems trained on an order of magnitude more, and data has become a binding constraint as much as computation. Chinchilla's analysis showed that, for a fixed training budget, many models of the period were too large for their data and that training smaller models on more tokens was more efficient 3.

Copyright and consent are unresolved. Much of the text was collected without the knowledge of its authors, and whether that constitutes fair use or infringement differs by jurisdiction. Some providers now publish summaries of training data or offer opt-outs; others disclose little.

Multilingual balance is uneven. English and a few other high-resource languages dominate, and performance in low-resource languages is correspondingly weaker.

Pretraining is self-supervised: the model is shown text and asked to predict the next token, repeatedly, over the corpus. No labels are needed, which is what makes the scale possible. The result completes text but does not follow instructions 2.

Instruction tuning then trains the model on examples of instructions and good responses, usually written or reviewed by people. Preference optimisation goes further: people rank candidate answers, a reward model learns from those rankings, and the model is adjusted to produce answers that rank higher, a technique popularised as reinforcement learning from human feedback 4. Variants replace the reward model with direct preference optimisation or with rule-based rewards for verifiable tasks such as mathematics.

Each stage changes behaviour. Much of the model's knowledge comes from pretraining; its manner of answering — brevity, refusal, formatting, tone — is largely a product of the later stages. Alignment therefore shapes what the model says without adding much to what it knows, which is why a helpful assistant can still be confidently wrong.

Safety training is layered on top: supervised examples of refusals, red-team prompts, and classifiers that screen inputs and outputs. Providers describe these measures in system cards, though the details are rarely reproducible from outside.

Language models perform well on tasks that can be specified in language and judged from text: summarisation, rewriting, translation, classification, extraction, question answering, and code generation. Performance improves with size and with the quality of prompting, and many tasks need no task-specific training at all 2.

In-context learning is the distinctive capability. A handful of examples in the prompt, or a plain instruction, is often enough to specify a task that would previously have required a trained model and a labelled dataset. Chain-of-thought prompting, which asks the model to write intermediate steps, improved performance on arithmetic and multi-step reasoning benchmarks when it was introduced 5.

Benchmarks show steady gains. Models exceed average human scores on several standard examinations, though such results say more about the tests than about general competence, and contamination of test items into training data is a persistent problem.

Multimodal models extend the pattern to images, audio, and video, and long-context variants accept hundreds of thousands of tokens. Capability is uneven across languages, domains, and formats, and a strong average score can conceal weak performance on the subset that matters to a particular user. The same model can pass an examination and fail to follow a two-line instruction, and which of the two it does depends on phrasing as much as on ability.

Reasoning is the most contested capability. Models can solve multi-step problems when they are allowed to generate intermediate tokens, and a generation of reasoning models trained with reinforcement learning on verifiable problems spends more computation at inference time before answering 5. That approach improved results on mathematics, coding, and some scientific questions, at higher latency and cost.

Tool use extends the model beyond its weights. A model can call a search engine, a calculator, a code interpreter, or a database, and incorporate the result into its answer. Retrieval over a document collection grounds answers in sources that can be checked, which reduces but does not eliminate fabrication.

Agentic use chains these calls into a loop: plan, act, observe, revise. Reliability falls as the number of steps grows, because each step can introduce an error that later steps build on. Systems that work impressively in a demonstration often fail on the long tail of real tasks.

Hallucination is the most discussed failure: the model produces a confident statement that is false or unsupported, because it is optimising for plausible text rather than for truth 6. Surveys of the phenomenon distinguish fabrication from unfaithfulness to a provided source, and note that retrieval, citation, and uncertainty training reduce the rate without eliminating it.

Context is finite. A model attends to a bounded window of tokens, information outside it is invisible, and performance degrades on very long inputs even within the window. Attention over long sequences is also expensive, so providers trade context length against price.

Reasoning is uneven: multi-step arithmetic, planning, and problems requiring exact bookkeeping are weaker than the fluency suggests. Models are sensitive to prompt wording, order of options, and irrelevant context, and they can be sycophantic, agreeing with a stated premise even when it is false.

Training data introduces bias, and outputs can reproduce it. Memorisation of rare text raises privacy questions, and the energy cost of training and serving is substantial. None of these limitations has a general fix; each is mitigated by a combination of data work, training, tools, and product design.

Training a frontier model requires thousands of accelerators running for weeks, and reported costs run to tens of millions of dollars or more, before data, staff, and failed runs are counted. Serving is cheaper per query but adds up at scale, and latency and price shape which applications are viable.

Scaling studies found smooth improvements in loss as parameters, data, and computation grew together, which guided several years of investment 7. Chinchilla refined the recipe by showing that data and parameters should grow in proportion, and that many earlier models were under-trained 3. Whether the returns continue at the same rate as models grow larger is contested, and the data constraint suggests they cannot continue indefinitely on the current recipe.

Access takes three broad forms: a hosted API, downloadable weights, and a consumer product. Hosted access keeps the model under the developer's control but makes the provider a gatekeeper; open weights allow inspection, local deployment, and fine-tuning but shift responsibility to the user. Distillation and quantisation let smaller models inherit much of a larger model's behaviour at a fraction of the cost.

Evaluation is immature. Benchmarks saturate, contamination is hard to rule out, and behaviour depends on prompting, on the version served, and on the tools attached. Evaluators increasingly test for specific risks, such as cyber capability, persuasion, or dangerous knowledge, rather than for general ability, and they compare models against human baselines on tasks with verifiable answers.

Policy has followed. The European Union's AI Act imposes duties on general-purpose models: technical documentation, a policy on copyright compliance, a summary of training content, and, for models above a compute threshold, systematic risk assessment and incident reporting 8. The obligations took effect in stages from 2025, and a code of practice was drafted to describe how compliance can be shown.

Other jurisdictions have taken different routes. Some regulate uses rather than models, some require disclosure for chatbots and synthetic media, and some have no dedicated regime at all. Providers argue that compliance costs entrench large firms; regulators reply that the costs are proportionate to the risks and that smaller developers face lighter duties.

The practical consequence for users is that fluent prose should be treated as a draft rather than a finding, and that any claim which matters should be checked against a source the model did not generate.

Critics make several distinct arguments. The first is that a system trained to predict text learns the statistical regularities of language without any model of what the words refer to, and that describing its output as reasoning or understanding overstates what it does. The phrase stochastic parrot, coined in 2021, summarised that position before the current generation existed.

The second is that benchmark gains are a poor proxy for usefulness. Contamination, prompt sensitivity, and the gap between multiple-choice tests and real tasks mean that a model can improve on paper while remaining unreliable in practice. Evaluators have responded with harder, more realistic tests, but the incentive to optimise the benchmark remains.

The third is political. Training data is collected at scale without consent, the resulting models are owned by a few firms, and the labour that produced the data — writing, translation, annotation, moderation — is poorly paid and increasingly automated. The fourth is environmental: data centres consume water and electricity, and the local costs fall on the communities that host them.

Defenders reply that the capabilities are real and useful, that the criticisms describe risks to be managed rather than reasons to stop, and that the alternative — leaving the technology to those who ignore the risks — is worse. Both positions are represented in the policy debate, and neither has been settled by evidence. The European Union's general-purpose model rules are an attempt to make at least some of the claims auditable, by requiring documentation that outside evaluators can inspect 8.