Reinforcement Learning

Reinforcement learning (RL) is a branch of machine learning in which an agent learns to make decisions by interacting with an environment. At each step, the agent observes the current state of the environment, takes an action, receives a reward signal, and transitions to a new state. Over time, the agent learns a policy, a mapping from states to actions, that maximizes the expected cumulative reward.

The fundamental distinction from supervised learning is that in reinforcement learning, there are no labeled correct answers. The agent must discover effective behaviors through exploration and feedback from the environment. This makes RL well-suited for sequential decision problems where the right action depends on context and where the consequences of actions unfold over time.

Historical reinforcement learning successes include game-playing agents like AlphaGo (Go), OpenAI Five (Dota 2), and AlphaStar (StarCraft II), which achieved superhuman performance by playing millions of games and refining their policies through self-play.

In large language model training, reinforcement learning is used in the RLHF pipeline. The language model plays the role of the agent, generating text is the action, and the reward model provides the reward signal by scoring how much a human would prefer the generated text. Proximal Policy Optimization (PPO) is the RL algorithm most commonly used to update the language model parameters in response to reward signals.

Understanding reinforcement learning helps AI trainers understand why human feedback has such a direct and lasting effect on model behavior: it is literally the reward signal in an RL loop.