Precision and Recall for Labelers
Precision measures how many of an annotator's positive labels were correct; recall measures how many true positives they actually caught.
Precision and recall are the two core axes for measuring labeling quality against a known-correct reference, usually a gold set. Precision answers "of the items you labeled as X, how many actually were X?" while recall answers "of all the items that actually were X, how many did you catch?" An annotator can maximize one at the expense of the other — labeling everything as positive gets perfect recall and terrible precision, labeling almost nothing gets the reverse.
These metrics matter more than a single aggregate accuracy score, especially under class imbalance: an annotator could have 99% accuracy on a rare-positive-class task just by labeling everything negative, while missing every real case.
Most QA systems track both together, sometimes combined into an F1 score, and use the split to diagnose whether an annotator is overly cautious (low recall) or overly aggressive (low precision) rather than just uniformly wrong.
What this means for trainers
If feedback tells you you're "missing cases" that's a recall problem; if it tells you you're "over-flagging" that's a precision problem — the fix for each is the opposite of the other, so knowing which one you're being told matters.
Related terms
Put this into practice
Browse open AI training roles from Alignerr, Mercor, Outlier, and more.
Browse AI training jobs