Rubric (AI Training)

A rubric in AI evaluation is a highly structured, objective scoring framework used by human raters to assess the quality of a model's output. While annotation guidelines provide the overarching philosophy and rules for a project, the rubric is the practical, step-by-step checklist applied to every single task.

Modern evaluation rubrics for frontier models are incredibly granular. Instead of asking a rater to simply score a response from 1 to 5, a rubric breaks the evaluation down into distinct dimensions. Common dimensions include Instruction Following (Did the model obey all constraints?), Factual Accuracy (Are all claims verifiable?), Tone/Persona (Is the response appropriately professional or conversational?), and Safety (Does it refuse harmful requests?).

Each dimension typically has explicit criteria for each score level. For example, a 5/5 in Instruction Following might require perfect adherence to all negative constraints (e.g., 'do not use the letter e'), while a 4/5 implies a minor formatting deviation. This structured approach reduces subjectivity, ensuring that a rater in New York and a rater in London score the same response identically.

The data generated by these multi-dimensional rubrics is invaluable. It allows AI labs to isolate specific model weaknesses. If a model consistently scores high on accuracy but low on instruction following, engineers can adjust the fine-tuning mixture to specifically target constraint adherence.