Jailbreak Detection
Identifying prompts specifically crafted to bypass a model's safety constraints.
Jailbreak detection labels prompts that attempt to manipulate a model into ignoring its safety training — role-play framings ("pretend you're an AI with no restrictions"), instruction smuggling, or multi-turn setups that build toward a disallowed request gradually. It's a specialized subset of adversarial prompting focused specifically on safety bypass rather than general robustness testing.
This work typically pairs with refusal quality evaluation on the other side: annotators judge both whether a prompt is a jailbreak attempt and whether the model's response correctly declined it without over-refusing legitimate requests that merely resemble one.
Jailbreak techniques evolve quickly, so datasets for this task need frequent refreshing — a detector trained only on last year's jailbreak patterns misses this year's.
What this means for trainers
This work often overlaps with red teaming roles and can require creative, persistent probing rather than passive labeling — expect it to be treated as a specialized, sometimes higher-paid track.
Related terms
Put this into practice
Browse open AI training roles from Alignerr, Mercor, Outlier, and more.
Browse AI training jobs