Summarization Evaluation

Summarization evaluation judges a generated summary along at least three independent axes: faithfulness (does it avoid claims not supported by the source — closely related to groundedness), coverage (does it capture the important points, not just any points), and clarity (is it concise and readable on its own).

A summary can fail on any one axis while succeeding on the others — a summary can be perfectly faithful and clear while missing the single most important point of the source, or comprehensive and accurate while being nearly as long as the original text it's meant to condense.

This evaluation type is a specific instance of rubric-based evaluation, and is one of the more commonly outsourced annotation tasks since it requires careful reading comprehension but not deep domain expertise.