Code Correctness Evaluation

Code correctness evaluation goes beyond checking that generated code compiles or looks reasonable — it verifies the code actually does what was asked, typically by running it against test cases, tracing through edge-case inputs, or checking it against a reference implementation. This is distinct from code review annotation, which focuses on style, security, and maintainability rather than whether the code runs correctly.

This evaluation type demands real domain expertise: an annotator without programming background can be fooled by code that looks idiomatic and confident but silently produces wrong output on an untested input path, which is exactly the kind of failure this task exists to catch.

Code correctness datasets are a core input to coding-focused model evaluation and RLHF, since raw pass/fail test results give a much cleaner training signal than subjective preference alone.