The name comes from "freeing" a phone from its manufacturer's restrictions. In language models, the restrictions are those installed by RLHF and the system prompt: refusing to give dangerous instructions, not revealing one's own rules. A jailbreak is a text that manages to get around them. Early versions were handcrafted: asking the model to play a character "with no rules," wrapping the request in a fiction, splitting it into innocent-looking steps.
Zou et al. (2023) showed that the process can be automated. With a search combining gradients and greedy choices on open models, they found nonsensical-looking suffixes that, appended to a forbidden request, maximize the probability that the model begins with an affirmative answer. The unsettling part was transfer: suffixes optimized on small models also worked against the public interfaces of ChatGPT, Bard and Claude. It is a textual relative of adversarial examples in vision (adversarial training).
Why do they work? Millière (2025) offers a philosophical explanation: current alignment installs shallow behavioral dispositions (be helpful, be honest, do no harm) without a genuine capacity to deliberate when those norms conflict. Jailbreaks exploit precisely such conflicts, for instance by presenting a harmful request as a case in which refusing would be unhelpful or dishonest. Humans resist these traps better because they can reason about the norms themselves.
A variant, indirect prompt injection, hides instructions in data the model is going to read, such as a web page consulted by an agent (Greshake et al., 2023). A logician will recognize the problem: the system does not reliably distinguish between using a text as an instruction and mentioning it as data. Everything that enters the context window lies on the same plane, which is why no quotation marks are entirely safe. See Sydney.