Code Correctness Evaluation

Code correctness evaluation goes beyond checking that generated code compiles or looks reasonable. It verifies the code does what was asked, typically by running it against test cases, tracing through edge-case inputs, or checking it against a reference implementation. This is distinct from code review annotation, which focuses on style, security, and maintainability rather than whether the code runs correctly.

This evaluation type demands real domain expertise. An annotator without a programming background can be fooled by code that looks idiomatic and confident but silently produces the wrong output on an untested input path, which is exactly the kind of failure this task exists to catch.

Code correctness datasets are a core input to coding-focused model evaluation and RLHF, since raw pass or fail test results give a much cleaner training signal than subjective preference alone. A response that reads well but fails its test cases still needs to be marked as incorrect, regardless of how confident or well-formatted it appears.