Annotation

Annotation is the act of marking up raw data with information that makes it interpretable and usable for machine learning. It encompasses a wide range of tasks: labeling images with object categories, transcribing audio recordings, tagging text with named entities, rating the quality of AI responses, or marking the start and end positions of an answer span in a document.

Annotation differs from data collection in that the raw data already exists; annotation adds meaning to it. A recording of a phone call is raw data. Adding a transcript, a sentiment label, and a category (support request, complaint, compliment) is annotation. The same raw data can be annotated in multiple ways for different downstream tasks.

Annotation quality determines model quality. Models trained on well-annotated data learn accurate decision boundaries. Models trained on noisy or inconsistent annotations learn the noise as well as the signal. This is why annotation is not a commodity: skilled, calibrated annotators who understand the task deeply produce data that is worth far more per label than high-volume, low-expertise labeling.

In the context of large language model development, annotation has expanded to include preference annotation (comparing outputs and selecting the better one), evaluation annotation (rating outputs on multiple quality dimensions), and red-teaming annotation (identifying and documenting model failures). These tasks require reading comprehension, critical thinking, and often domain expertise.

Annotation is the profession that connects human judgment to machine learning at scale, and it sits at the foundation of every commercially deployed AI system in use today.