Pairwise Ranking
Comparing exactly two candidate outputs and choosing the better one.
Pairwise ranking is the simplest and most common form of preference ranking: two responses to the same prompt, one judgment call for which is better. Its simplicity is the point — humans are much more reliable at comparing two things directly than at scoring items independently on an absolute scale, since a comparative judgment doesn't require agreeing on what a "7 out of 10" means in the abstract.
Large-scale preference datasets for RLHF and DPO are built almost entirely from pairwise comparisons rather than absolute scores, precisely because pairwise judgments are more consistent across annotators and over time.
When more than two candidates need ranking, systems typically decompose the problem into multiple pairwise comparisons and aggregate them (e.g., via a tournament or Elo-style system) rather than asking annotators to sort a full list at once.
What this means for trainers
Resist the urge to rate absolute quality in your head — the task is comparative: which of these two is better, not how good is each one on its own.
Related terms
Put this into practice
Browse open AI training roles from Alignerr, Mercor, Outlier, and more.
Browse AI training jobs