safety and alignment

Jailbreak Detection

Identifying prompts specifically crafted to bypass a model's safety constraints.

What this means for trainers

This work often overlaps with red teaming roles and can require creative, persistent probing rather than passive labeling — expect it to be treated as a specialized, sometimes higher-paid track.

Jailbreak detection labels prompts that attempt to manipulate a model into ignoring its safety training, including role-play framings, instruction smuggling, or multi-turn setups that build toward a disallowed request gradually across several exchanges. It is a specialized subset of adversarial prompting focused specifically on safety bypass rather than general robustness testing.

This work typically pairs with refusal quality evaluation on the other side of the pipeline. Annotators judge both whether a prompt is a jailbreak attempt and whether the model's response correctly declined it without over-refusing a legitimate request that merely resembles one on the surface.

Jailbreak techniques evolve quickly, so datasets for this task need frequent refreshing. A detector trained only on last year's patterns misses this year's, which is part of why this category of work overlaps so heavily with ongoing red teaming effort rather than a fixed labeling task that gets finished once.