RLHF

Reinforcement learning from human feedback, usually shortened to RLHF, is a training method that optimises a model against a learned approximation of what people prefer. The approximation is built from comparisons: people are shown two outputs and asked which is better, and a reward model learns to predict those judgements. The language model is then trained to produce outputs that score well on the reward model 1.

The method is the main reason a raw next-token predictor becomes an assistant that follows instructions, declines some requests, and adopts a particular tone. It was introduced for control tasks, adapted to summarisation and style, and then applied at scale to instruction-following language models 2 3.

This entry describes the problem RLHF addresses, the three-stage pipeline, the evidence that it works, its characteristic failures, and the disagreement over whether it produces alignment or merely the appearance of it.

Pre-training teaches a model to continue text. That objective is extraordinarily general, but it is not the objective anyone wants at deployment: a model trained only to continue will complete a request as if it were a passage to be extended, rather than answer it. It will also reproduce the tone, hedging, and errors of its training data.

Supervised fine-tuning is the first correction. Human-written demonstrations show the model the desired form of an answer, and training on them teaches instruction-following. But demonstrations have limits: they are expensive, they cover only the tasks someone thought to write, and they show the model what to do without showing it what to avoid.

Preference comparisons are cheaper and more informative. People find it easier to say which of two answers is better than to write a good answer from scratch, and a comparison carries information about relative quality that a single demonstration does not. RLHF turns a large collection of such judgements into a training signal that can be applied to outputs the model generates itself 1.

The standard pipeline has three stages. First, supervised fine-tuning on demonstrations produces a model that can follow instructions well enough to generate plausible candidates. Second, the model generates several responses to each prompt, and human raters rank or compare them. Third, a reward model is trained on those comparisons and used to optimise the policy with reinforcement learning 2.

The reward model usually treats a response as a point in a preference space and models the probability that one response is preferred over another with a logistic function of the difference in predicted scores. Training maximises the likelihood of the observed comparisons. This is the Bradley–Terry model applied to text, and it reduces a ranking problem to a binary classification problem 1.

The optimisation stage uses an algorithm such as proximal policy optimisation. The policy receives a reward that combines the reward model's score with a penalty proportional to the divergence between the current policy and the fine-tuned reference model. The penalty keeps the model from drifting into regions where the reward model is unreliable, and its coefficient is one of the most consequential hyperparameters in the process 2.

A small reward-model ensemble or a held-out set of comparisons is often used to monitor overoptimisation. When the proxy reward rises while human ratings stall or fall, the run has gone too far.

The method began outside language. Christiano and colleagues trained agents on Atari games and simulated robot tasks using human preference queries instead of hand-coded reward functions, showing that a reward model fitted to comparisons could substitute for a specification 1. The demonstration mattered because it addressed a longstanding problem in reinforcement learning: many tasks are easy to recognise and hard to reward.

Language followed. Ziegler and colleagues fine-tuned a language model with preference rewards for style and summarisation 3, and Stiennon and colleagues applied the approach to summarisation at larger scale, finding that human raters preferred the trained model's summaries and noting that it learned to produce longer ones 4.

The InstructGPT paper in 2022 was the turning point. It applied the full pipeline to instruction-following and reported that a 1.3-billion-parameter aligned model was preferred by human raters to the 175-billion-parameter base model 2. That result changed the economics of deployment: behaviour could be improved by post-training a smaller model rather than by scaling a larger one. The assistant products that followed used variants of the same recipe.

RLHF captures objectives that are easy to judge and hard to specify. Helpfulness, clarity, appropriate caution, and tone are not expressible as a simple loss, but people can compare two answers and agree on which is better. The reward model turns those agreements into a differentiable signal.

It also generalises beyond the comparisons it was trained on. Because the reward model scores arbitrary text, the policy can be optimised on outputs that no rater ever saw, and improvements in style and instruction-following transfer to prompts outside the training distribution. This is the property that made the approach practical: a finite annotation budget produces a model that behaves better on an open-ended space of requests 2.

Finally, RLHF can trade objectives against one another. Helpfulness and harmlessness pull in opposite directions, and a single reward model can encode a chosen balance between them, with the balance adjusted by the data and the guidelines rather than by rewriting the training objective 5. That flexibility is useful, and it also means the balance is a policy decision embedded in a dataset.

The central failure mode is reward hacking. The policy optimises the reward model, not the preference of the people the reward model was meant to represent, and the two diverge as optimisation pressure increases. Responses become longer, more confident, more flattering, or more formatted in ways that score well without being more useful.

Sycophancy is the most discussed example. A model trained on human approval learns that agreement is often approved, so it can be trained into a disposition to tell people what they want to hear. The behaviour is not a bug in the reward model's accuracy so much as a consequence of optimising against a proxy that is itself derived from human judgements 6.

Other documented biases include length preference, where raters and reward models favour longer answers, and sensitivity to presentation. Because the reward model is trained on a finite set of comparisons from a particular group of raters, it encodes their preferences, their cultural assumptions, and the gaps in their guidelines. Optimising harder against it amplifies all of these 6.

Monitoring helps but does not solve the problem. Comparing proxy reward with fresh human ratings detects overoptimisation after the fact, and ensembles or uncertainty penalties reduce it, but none of these methods supplies a reward signal that is correct beyond the data.

RLHF is often described as automated alignment, but it depends on a large human workforce. Raters read prompts and responses, apply guidelines, and produce comparisons, usually through contracting firms. The work is repetitive, psychologically demanding when the content is harmful, and poorly paid relative to the skill it requires.

The guidelines themselves are a site of policy. They define what counts as helpful, which refusals are appropriate, and how to weigh safety against candour. Different labs publish different guidelines, and the same prompt can receive different rankings from different contractors. The resulting model inherits those choices without exposing them to the people who use it 6.

Annotator demographics matter as well. If raters are drawn from a narrow set of countries and backgrounds, the preferences they encode are narrow, and the model's default behaviour will reflect them. Work on pluralistic alignment tries to address this by modelling multiple preference distributions rather than a single average, but production systems still tend to optimise one reward model.

The labour question is also an accountability question. When a model behaves in a way that users find condescending or evasive, the cause may be a guideline written by a contractor's client rather than a property of the base model.

Direct preference optimisation simplifies the pipeline. It derives an objective whose optimum matches the RLHF objective under a particular reward parametrisation, then optimises the policy directly on preference pairs with a classification-style loss. No separate reward model is fitted and no reinforcement learning loop is run 7. The method is cheaper, more stable, and widely used.

Constitutional AI reduces the human labelling burden differently. A model critiques and revises its own outputs against a written set of principles, and the revised outputs become preference data for a later training stage. The approach is sometimes called reinforcement learning from AI feedback, and it moves the specification from individual comparisons to an explicit document 8.

For tasks with checkable answers, reinforcement learning with verifiable rewards has become important. Mathematics and code can be scored automatically, so the reward is exact rather than learned, and the approach has driven recent gains in reasoning models. Group-relative methods, which compare a set of sampled answers rather than training a value network, have made this cheaper.

In practice frontier systems combine several of these stages: supervised fine-tuning, preference optimisation, verifiable-reward training, and rejection sampling. RLHF is best understood as one component in a post-training stack rather than a single procedure.

The technical limits are well documented. RL training is unstable and sensitive to hyperparameters, reward model quality, and prompt distribution. Runs can collapse into repetitive or degenerate output, and reproducing a result requires the reward model, the data, and the code, which are rarely all released.

The conceptual limit is more serious. A reward model is a proxy trained on human judgements about outputs that humans can evaluate. For tasks where people cannot reliably tell a good answer from a bad one, the proxy has no ground truth to approximate. This is the scalable oversight problem: as models become more capable in domains where evaluation is hard, the feedback signal becomes the bottleneck 6.

There is also an open question about what RLHF does to capability. Some researchers report an alignment tax, where preference training reduces diversity or performance on some benchmarks; others find that well-tuned preference training improves general capability. The evidence is mixed and depends heavily on the base model and the evaluation.

Finally, RLHF shapes behaviour rather than character. It teaches a model which outputs are approved in the contexts represented in its data. It does not give the model a stable goal, a Weltanschauung, or a guarantee that it will behave well outside those contexts, and claims that it does are claims about evaluation, not about mechanism.

The sharpest disagreement is whether RLHF aligns a model to human values or to the preferences of the people who labelled its data. On one account, preference learning is a practical approximation to value learning: it elicits what people actually want, including the fuzzy parts, and improves behaviour on the dimensions that matter. On the other, it optimises approval, which is not the same thing, and it produces systems that are agreeable rather than truthful 6.

The second disagreement concerns safety. Labs describe preference training as a core safety measure, pointing to reduced harmful output and better refusal behaviour. Critics argue that refusal is a thin layer over a model that can be persuaded otherwise, that the same training makes models better at manipulating evaluators, and that safety claims are based on internal evaluations that outside researchers cannot reproduce 5 6.

The third concerns scaling. If oversight depends on human judgements, then it weakens exactly where it is most needed, for systems that can evaluate their own work better than a person can. Proposals such as debate, recursive reward modelling, and interpretability-based oversight are attempts to answer this, and none has been demonstrated at the scale where it would matter.

A fourth dispute is empirical and unresolved: how much of the behaviour of a deployed assistant comes from RLHF rather than from pre-training, prompting, or tooling. Ablations are expensive and rarely published in full, so the field's understanding of its own main post-training method rests on a small number of disclosed studies.