RLHF (Reinforcement Learning from Human Feedback)

Reinforcement Learning from Human Feedback (RLHF) is the dominant technique used to align large language models with human values and intentions. The process has three distinct stages. First, a base language model is fine-tuned on a highly curated set of human-written demonstrations, producing a supervised baseline (Supervised Fine-Tuning or SFT). Second, human raters are given pairs of model outputs generated from the same prompt. They compare these outputs and indicate which response is better, often using detailed rubrics that score on accuracy, helpfulness, tone, and safety. These comparisons are used to train a separate 'reward model' that learns to mathematically predict human preferences.

Third, the language model is updated using a reinforcement learning algorithm—typically Proximal Policy Optimization (PPO)—that maximizes the reward model's score while applying a KL penalty to stay close to the supervised baseline, ensuring the model doesn't degenerate into spewing nonsense just to game the reward model.

The quality of the final model depends entirely on the quality of the human preference data. If raters misunderstand the annotation guidelines, apply criteria inconsistently, or exhibit systemic biases (like preferring longer but hallucinated responses over concise, factual ones), the reward model will learn and reinforce these flaws—a phenomenon known as reward hacking.

RLHF is the backbone behind nearly every major commercial AI assistant, including ChatGPT, Claude, Gemini, and Llama. Variants such as Direct Preference Optimization (DPO) and RLAIF have emerged to reduce the computational cost or dependence on massive human annotation pipelines, but human judgment remains the gold standard for defining the initial preference distribution.