evaluation

Instruction Following Evaluation

Checking whether a model's output satisfies the explicit constraints given in the prompt.

Instruction-following evaluation isolates one specific question: did the model do what it was actually asked, including every constraint — word count, format, tone, list length, required sections — not just the general topic. A response can be well-written, accurate, and still fail this evaluation if it ignores a stated constraint like "answer in exactly three bullet points."

This differs from broader helpfulness scoring in that it's checklist-driven rather than holistic: annotators typically extract every explicit and implicit constraint from the prompt and verify each one independently before forming an overall judgment.

Models are notoriously inconsistent at following compound instructions (multiple constraints at once), which makes this one of the more diagnostic evaluation types for catching real capability gaps.

What this means for trainers

Read the prompt for constraints first, response second — annotators who skim the prompt and focus only on response quality routinely miss the exact failures this task exists to catch.