safety and alignment ml fundamentals high-volume term

Adversarial Examples

Carefully crafted inputs designed to cause machine learning models to produce incorrect outputs, often through imperceptible modifications.

Adversarial examples are inputs to a machine learning model that have been intentionally perturbed or manipulated by an attacker to cause the model to make a mistake, while appearing completely normal or benign to a human.

In computer vision, a classic adversarial example involves changing a few specific pixels in an image of a stop sign. To a human, it still clearly looks like a stop sign, but the subtle pixel noise tricks the autonomous vehicle's neural network into classifying it as a speed limit sign, with potentially disastrous consequences.

In the realm of Large Language Models, adversarial examples take the form of carefully crafted text strings. Researchers have discovered that appending specific, seemingly random sequences of characters (e.g., 'Zingo bango bongo !?%') to a prompt can completely shatter a model's safety alignment, causing it to ignore all RLHF training and output toxic or harmful content. These specific strings exploit vulnerabilities in the model's high-dimensional vector space, forcing the attention mechanism to focus on the wrong features.

Discovering and patching these adversarial examples is a major focus of AI security research, as they prove that deep neural networks learn representations of the world that are fundamentally alien to human perception.

What this means for trainers

During highly technical red-teaming tasks, you may be asked to generate or test adversarial examples. This involves systematically altering the phrasing, syntax, or encoding of a prompt to find the exact linguistic trigger that causes the model's safety filters to fail, while keeping the prompt semantically identical to a human reader.