Adjudication
The process of resolving conflicting labels from multiple annotators into a single, final decision.
Adjudication is the step in an annotation pipeline where disagreements between labelers are resolved into one canonical answer. When two or more annotators label the same item differently under inter-annotator agreement tracking, that disagreement doesn't get averaged away — it gets routed to an adjudicator: a senior reviewer, a lead annotator, or a small committee empowered to make the final call.
Good adjudication does more than pick a winner. It documents why the losing label was wrong, and that reasoning often feeds back into the annotation guidelines so the same disagreement doesn't recur. Teams that skip this feedback loop end up re-adjudicating the same edge cases over and over.
Adjudication is typically reserved for high-disagreement or high-stakes items rather than every task, since it's slower and more expensive than first-pass labeling.
What this means for trainers
On rubric-heavy platforms, being selected as an adjudicator is a trust signal and often pays a premium rate — it means your judgment on ambiguous cases is considered reliable enough to overrule other annotators.
Related terms
Put this into practice
Browse open AI training roles from Alignerr, Mercor, Outlier, and more.
Browse AI training jobs