Gold Set
A verified benchmark set of pre-labeled items used to audit annotator quality against a known-correct answer.
What this means for trainers
Assume every task could be a gold item. Platforms rarely disclose gold-set accuracy directly, but sudden changes in task availability or pay tier are often downstream of it.
A gold set is a collection of items whose correct labels are already known, usually established by expert reviewers or through adjudication, and quietly mixed into an annotator's regular task queue. Because the annotator cannot tell a gold item from a normal one, their accuracy on the gold set is a clean, unbiased measure of their true quality rather than a measure influenced by knowing they are being tested.
Gold sets are distinct from a general gold standard. Where a gold standard is a loose reference to the best available answer, a gold set is a specific, versioned dataset built for auditing. It is the backbone of automated quality monitoring, since platforms can compute an annotator's gold-set accuracy in real time and gate access to higher-paying tasks behind a minimum score.
Gold sets need periodic refreshing. If annotators figure out which items are gold, whether through repetition or a leaked pattern in how gold items are formatted, the measurement stops being honest and the platform loses its cleanest quality signal until the set is rebuilt.
Related terms
Put this into practice
Browse open AI training roles from Alignerr, Mercor, Outlier, and more.
Browse AI training jobs