Adjudication

Adjudication is the step in an annotation pipeline where disagreements between labelers are resolved into a single canonical answer. When two or more annotators label the same item differently under inter-annotator agreement tracking, that disagreement is not averaged away. It gets routed to an adjudicator, which might be a senior reviewer, a lead annotator, or a small committee empowered to make the final call.

Good adjudication does more than pick a winner. It documents why the losing label was wrong, and that reasoning often feeds back into the annotation guidelines so the same disagreement does not recur. Teams that skip this feedback loop end up re-adjudicating the same edge cases over and over, spending review budget on problems that a single guideline update could have closed permanently.

Adjudication is typically reserved for high-disagreement or high-stakes items rather than every task, since it is slower and more expensive than first-pass labeling. Platforms usually route only items that fail an agreement threshold, or items flagged during ambiguity resolution, into the adjudication queue rather than sending every submission through a second layer of review.