training methods

Reward Model

A model trained on human preference comparisons that predicts which AI outputs humans consider better.

A reward model is a specialized neural network that acts as the automated judge during the Reinforcement Learning from Human Feedback (RLHF) pipeline. Its sole purpose is to take a prompt and a corresponding model response, and output a single scalar value (a 'reward score') representing how much a human would like that response.

Training a reward model is the critical bridge between human annotation and machine learning. First, human raters evaluate thousands of pairs of responses, indicating which one is better (e.g., Response A > Response B). The reward model is trained on this dataset using a ranking loss function. It learns the underlying patterns of human preference: it learns to assign high scores to responses that are factual, polite, and perfectly formatted, and negative scores to responses that are toxic, hallucinated, or unhelpful.

Once the reward model accurately mimics human judgment, it is used to train the actual language model. During the reinforcement learning phase (typically using PPO), the language model generates millions of responses. It doesn't need humans to rate them; instead, the reward model instantly scores them. The language model updates its weights to maximize the score it receives from the reward model.

If the reward model is flawed—due to biased or noisy human data—the language model will exploit those flaws, leading to a breakdown in alignment.

What this means for trainers

When you complete a preference comparison task, you are not training the main AI directly. You are training the reward model. You are teaching this proxy judge exactly what humans value, so that it can autonomously score millions of responses later. If your ratings are inconsistent, the reward model becomes confused and useless.