Neural network

A neural network is a machine-learning model built from layers of simple units, each computing a weighted sum of its inputs and passing the result through a nonlinear function. Individually the units are trivial; collectively, with enough layers and enough data, they approximate complicated functions and learn representations that nobody designed by hand.

The idea is old, proposed in the 1940s and refined in the 1980s, but its practical dominance dates from the 2010s, when graphics processors, large datasets, and better training methods made deep networks trainable at scale. Almost every system described as AI in the 2020s relies on one.

This entry covers the structure of neural networks, how they are trained, the main architectures, the hardware that runs them, what they are used for, and the limits and disputes that remain.

A single artificial neuron takes a vector of inputs, multiplies each by a learned weight, adds a bias, and applies an activation function. A network arranges such units in layers, input to hidden to output, with each layer feeding the next. The weights are the model's parameters, and training is the process of setting them.

The activation function is what makes depth useful. Without a nonlinearity, stacked layers collapse into one linear transformation, however many there are. Rectified linear units, which pass positive values and zero out negative ones, became standard because they train quickly and avoid some saturation problems of earlier sigmoid functions. Modern variants such as GELU and SwiGLU are smooth approximations with similar behaviour.

Architecture is the choice of which units connect to which. A fully connected layer lets every input influence every output; a convolutional layer restricts each unit to a local patch and shares weights across positions; an attention layer lets every position weigh every other. Those restrictions are priors about the data, and they matter more than the activation function does.

Width and depth determine capacity. A sufficiently wide network can approximate any continuous function on a bounded domain, a result known as the universal approximation theorem, but the theorem says nothing about whether the function can be learned from finite data or computed cheaply. That gap between expressibility and learnability is where most practical work happens 1.

Training minimises a loss function measuring the difference between the network's output and a target. Backpropagation computes the gradient of that loss with respect to every weight by applying the chain rule backwards through the layers, and an optimiser then updates the weights to reduce the loss 2.

The method was popularised in the 1980s and refined steadily. Stochastic gradient descent estimates the gradient from small batches rather than the whole dataset; momentum and adaptive methods such as Adam adjust the step size per parameter; dropout, batch normalisation, and residual connections made deeper networks easier to train; and learning-rate schedules decide how far to move at each stage.

Modern runs repeat this loop over enormous datasets, often for weeks on thousands of accelerators, and the resulting parameters number in the billions or trillions. The arithmetic is dominated by matrix multiplication, which is why the hardware that trains networks resembles the hardware that renders graphics.

Training is not the same as learning to generalise. A network can drive its training loss close to zero while memorising noise, which is why held-out data, augmentation, weight decay, and early stopping are part of the method rather than optional refinements.

Three families dominate. Convolutional networks apply learned filters over local patches of an image, exploiting the fact that nearby pixels are related and that a feature useful in one corner is useful elsewhere; a convolutional network won the ImageNet competition in 2012 and set off the deep-learning era 3. Recurrent networks, and later the long short-term memory design, processed sequences by carrying a hidden state forward in time, which made them the standard for speech and translation until attention displaced them.

The transformer, introduced in 2017, replaced recurrence with self-attention, letting every position in a sequence attend to every other in parallel 4. It trains far more efficiently on modern hardware and now underpins most language, vision, and multimodal systems. Its cost grows with the square of sequence length, which is why long-context methods are an active research area.

Two design ideas cut across families. Mixture-of-experts layers route each token to a small subset of parameters, so total capacity can grow faster than computation per token. Residual connections add a layer's input to its output, which keeps gradients flowing through very deep stacks and made networks with a hundred or more layers routine 5.

Feedforward networks remain the standard building block for tabular data, where rows are independent and the relationships are not spatial or sequential. Graph networks extend the same principles to relational data, and diffusion models, which learn to reverse a noising process, now dominate image and video generation.

The intellectual history runs through several distinct programmes. McCulloch and Pitts described threshold logic units in 1943 as a model of nervous activity 6, and Rosenblatt's perceptron of 1958 added a learning rule and a hardware implementation that could adapt its weights from examples 7.

Minsky and Papert's 1969 analysis showed that a single-layer perceptron could not represent simple nonlinear functions such as exclusive or, and the critique contributed to a long contraction in neural-network funding. Interest revived in the 1980s when backpropagation made multilayer training practical, and convolutional networks were applied to handwritten digit recognition before the end of the decade 2.

Progress stalled again in the 1990s and 2000s, when support vector machines and other statistical methods were easier to train and analyse. Two changes reversed the position: graphics processors made large matrix computations cheap, and internet-scale labelled datasets made overfitting less binding. The 2012 ImageNet result is conventionally treated as the turning point 3.

Since then the milestones have arrived quickly: residual networks in 2015, the transformer in 2017, large pretrained language models from 2018, and diffusion models for images from 2020. Each depended on the previous one's architecture, data, or hardware rather than on a single breakthrough.

Depth lets a network build hierarchical representations: edges into shapes, shapes into objects, objects into scenes. In language models, early layers appear to capture surface patterns and later layers more abstract structure, though the interpretation of individual units remains unreliable.

Depth also buys representational efficiency. A deep network can carve its input space into vastly more linear regions than a shallow one with the same number of units, which is the formal reason depth is worth the training difficulty.

Scale amplifies all of this. Larger networks trained on more data generally perform better on benchmarks, a regularity formalised in scaling studies and followed, with diminishing returns and occasional surprises, by successive model generations. Parameters per model rose from millions in the 2000s to hundreds of billions, and mixture-of-experts designs pushed totals past a trillion while keeping the computation per token manageable 1.

Some claimed benefits of scale are contested. Analyses of benchmark curves have argued that several supposed emergent abilities are artefacts of the metrics chosen, appearing as discontinuous jumps where the underlying improvement is smooth. Whether scale alone yields qualitatively new capabilities is therefore an open empirical question rather than a settled result.

Neural networks are trained and served on graphics processing units or dedicated accelerators, because the dominant operation, matrix multiplication, parallelises well. The shift to these devices is part of why deep learning arrived when it did: commodity graphics chips turned a computation that had been impractical into one a research group could afford 8.

The hardware constraint is often memory rather than arithmetic. Large models must hold weights and intermediate activations in high-bandwidth memory, and moving data between memory and processors costs more energy than the arithmetic itself. That is why accelerator design for AI emphasises memory bandwidth, on-chip caches, and fast interconnects between devices.

Energy use is substantial. A single large training run consumes electricity comparable to the annual use of many households, and serving a popular model adds a continuing load that can exceed the training cost over the model's lifetime. Published efficiency figures are difficult to verify independently and depend on hardware, cooling, and the local energy mix.

Efficiency research is now as commercially important as frontier scale. Quantisation stores weights in eight or four bits, pruning removes parameters, distillation trains a small model to imitate a large one, and conditional computation avoids work that the input does not need 8.

Vision was the first large-scale success: classification, object detection, segmentation, and face recognition all moved to convolutional and then transformer backbones. Speech recognition and synthesis followed, as did machine translation, where sequence models replaced phrase-based systems within a few years.

Language is now the largest application. Networks trained to predict text serve as the substrate for assistants, coding tools, search, and document processing, and multimodal models extend the same architecture to images, audio, and video.

Recommendation systems use networks to learn embeddings of users and items, then rank candidates by predicted engagement. Scientific applications include protein structure prediction, weather forecasting, materials screening, and the analysis of particle-physics data, where networks often serve as fast approximations of expensive simulations 1.

Not every deployed system is a neural network. Scheduling, verification, and business-rule engines remain symbolic, and the most reliable systems usually combine learned components with explicit constraints.

Neural networks are opaque. Their parameters encode a function rather than a rationale, and explanation methods give approximations that can mislead. A saliency map shows which pixels changed the output, not why the model found them decisive.

They are also brittle. Small adversarial changes to an input can change an output sharply, and performance falls when test conditions drift from training conditions. A model trained on clear photographs may fail in rain, and a model trained in one hospital may learn the scanner rather than the disease.

They are data-hungry and compute-hungry, and they do not learn the way people do. A child learns many concepts from a few examples; a network typically needs thousands. Whether this reflects a difference in kind or only in efficiency is disputed, but it is the reason transfer learning and pretraining matter so much in practice 1.

Catastrophic forgetting is a related weakness. Updating a network on a new task tends to overwrite what it learned on the previous one, so continual learning remains an open problem. Training data also carries its biases into the model, and the model has no independent way to check them.

The strongest criticism is that neural networks are not models of anything. A brain does not obviously implement backpropagation, neurons are not differentiable units in a layered graph, and biological learning appears to rely on local signals and sparse spikes. Defenders reply that the goal is engineering, not neuroscience, and that the comparison to brains is a metaphor rather than a claim 6.

A second dispute concerns understanding. One position holds that a network's behaviour is exhausted by its input-output mapping, so questions about meaning are category errors. Another holds that systems trained on text alone cannot ground their representations in the world, and that fluent output therefore overstates what the network knows.

A third dispute is about research strategy. The argument sometimes called the bitter lesson holds that methods which scale with computation and learn from data consistently beat methods that encode human knowledge, and that the field should stop building structure in by hand. Critics answer that sample efficiency, interpretability, and robustness are not problems that more computation has solved, and that the lesson is really about benchmark design.

A fourth concerns political economy: training large networks concentrates capability in firms that can afford accelerators and data, and the environmental cost of that training falls on the regions that host the data centres.