Reinforcement Learning through Human Feedback (RLHF): aligning models with human preferences
Once a language model has learned how to structure answers through instruction tuning, a second, equally important step follows: shaping its responses according to human judgement. This process, known as Reinforcement Learning through Human Feedback (RLHF), connects technical optimisation with normative and social expectations.

Even after instruction tuning, a model may still produce markedly different answers to the same prompt. This variability arises because language generation is inherently probabilistic: several responses may be equally plausible, each receiving a similar likelihood score. RLHF addresses this by teaching the model which of these alternatives humans perceive as preferable. Crucially, this preference is not determined by factual accuracy alone, but by broader criteria such as clarity, tone, relevance, and sensitivity to context.

Additional explanation is provided in the clip below:

How RLHF works

The RLHF procedure consists of several stages:

An instruction-tuned model is presented with a single prompt.
The model generates multiple candidate responses.
Human evaluators assess and rank these responses based on perceived quality.
These rankings are then used to apply an additional reinforcement learning step, adjusting the model’s parameters so that responses preferred by humans are more likely to be produced in future interactions.

Effects of RLHF
This feedback mechanism has several important consequences:

socially appropriate behaviour
First, it encourages the model to communicate in a more polite, coherent, and context-aware manner. Responses that are perceived as helpful, respectful, and considerate are consistently favoured over dismissive or aggressive alternatives, shaping the model’s interaction style accordingly.
safer and more responsible outputs
RLHF also plays a key role in improving safety and ethical reliability. Harmful, offensive, or discriminatory responses are systematically penalised, reducing the likelihood that such content is generated.
adaptation to stylistic preferences
Finally, the process helps the model adjust its tone and level of detail to situational expectations. If evaluators consistently prefer concise answers over lengthy explanations in a given context, the model learns to reflect this preference in its output.

In this way, Reinforcement Learning through Human Feedback integrates human values into the model’s decision-making process, alongside technical performance. This step is essential for deploying AI systems in a socially acceptable and responsible manner. At the same time, RLHF is not without limitations. Several of these will be discussed later in this module.