Summarization Evaluation

Summarization evaluation judges a generated summary along at least three independent axes: faithfulness, meaning whether it avoids claims not supported by the source and closely related to groundedness; coverage, meaning whether it captures the important points rather than just any points; and clarity, meaning whether it reads as concise and understandable on its own.

A summary can fail on any one axis while succeeding on the others. A summary can be perfectly faithful and clear while missing the single most important point of the source, or comprehensive and accurate while running nearly as long as the original text it was supposed to condense.

This evaluation type is a specific instance of rubric-based evaluation, and it is one of the more commonly outsourced annotation tasks since it requires careful reading comprehension but not deep domain expertise in a technical field.