Few-Shot Learning

Few-shot learning refers to the ability of a model to perform a new task given only a handful of labeled examples in the input context. For large language models, few-shot learning is demonstrated in-context: examples of the desired input-output mapping are included in the prompt, and the model generalizes from these examples to handle new inputs of the same type without any weight updates or retraining.

The capability was a notable discovery in large language models. GPT-3, released in 2020, demonstrated that sufficiently large models could perform tasks like translation, question answering, and arithmetic from just a few examples in the prompt, with no task-specific fine-tuning. This suggested that large models internalize meta-learning capabilities during pre-training: they learn how to learn from patterns.

Few-shot learning is most effective when the examples are representative of the task, clearly formatted, and diverse enough to convey the underlying pattern rather than surface features. Poorly chosen examples can mislead the model, causing it to generalize to the wrong pattern.

Few-shot prompting is distinct from fine-tuning: fine-tuning updates the model's weights using training examples, while few-shot learning uses examples only as context, with no weight updates. Fine-tuning is more compute-intensive but generally more reliable for very specific tasks; few-shot learning is flexible and instant but more variable in quality.

Many annotation guidelines use worked examples as few-shot demonstrations. Including a calibration example of a correct rating with a rationale is a form of few-shot instruction to the model of how to behave, as well as training for the rater.