Calibration
The process of aligning annotators on a shared interpretation of the guidelines before and during production work.
Calibration is what happens before a large annotation effort starts producing real data: a small group of annotators labels the same sample set, compares results, discusses disagreements, and converges on a shared interpretation of the guidelines. Only once agreement on this sample reaches an acceptable threshold does the team move to full production.
Calibration isn't a one-time event. Guidelines evolve, new edge cases surface, and annotator populations turn over — so mature pipelines re-run calibration periodically to catch drift before it degrades the dataset.
Without calibration, the first batch of production labels effectively becomes an uncontrolled experiment in how differently people read the same instructions.
What this means for trainers
Calibration rounds are usually unpaid or low-paid onboarding steps, but performing well in them is often what unlocks access to the better-paying production queue.
Related terms
Put this into practice
Browse open AI training roles from Alignerr, Mercor, Outlier, and more.
Browse AI training jobs