Gold Set

A gold set is a collection of items whose correct labels are already known — usually established by expert reviewers or adjudication — and quietly mixed into an annotator's regular task queue. Because the annotator can't tell a gold item from a normal one, their accuracy on the gold set is a clean, unbiased measure of their true quality.

Gold sets are distinct from a general gold standard: where "gold standard" is a loose reference to the best available answer, a gold set is a specific, versioned dataset built for auditing. It's the backbone of automated quality monitoring — platforms can compute an annotator's gold-set accuracy in real time and gate access to higher-paying tasks behind a minimum score.

Gold sets need periodic refreshing. If annotators figure out which items are gold (through repetition or leaked patterns), the measurement stops being honest.