Precision and Recall for Labelers
Precision measures how many of an annotator's positive labels were correct; recall measures how many true positives they actually caught.
What this means for trainers
If feedback tells you you're "missing cases" that's a recall problem; if it tells you you're "over-flagging" that's a precision problem — the fix for each is the opposite of the other, so knowing which one you're being told matters.
Precision and recall are the two core axes for measuring labeling quality against a known-correct reference, usually a gold set. Precision answers the question of how many of the items labeled as a given class belonged to that class, while recall answers how many of the items that truly belonged to that class were caught. An annotator can maximize one at the expense of the other: labeling everything as positive gets perfect recall and poor precision, while labeling almost nothing gets the reverse.
These metrics matter more than a single aggregate accuracy score, especially under class imbalance. An annotator could post a very high accuracy score on a rare-positive-class task simply by labeling everything negative, while missing nearly every real case that mattered.
Most QA systems track both together, sometimes combined into an F1 score, and use the split to diagnose whether an annotator is overly cautious, which shows up as low recall, or overly aggressive, which shows up as low precision, rather than treating both errors as the same problem.
Related terms
Put this into practice
Browse open AI training roles from Alignerr, Mercor, Outlier, and more.
Browse AI training jobs