Jailbreaking and what it teaches us about LLMs

On June 12, three days after releasing Fable 5, Anthropic took it offline. Chaos flooded my X feed. The drama was a preview of the inevitable collision between AI and policy, and at the center of it all was an odd technical term: jailbreaks.

Here’s Pliny the Liberator, a revered anon on X who often jailbreaks models within hours of release:

“Long-context reference tracking” and “Parseltongue-style text transforms”. Sounds fun! But what does it all mean?

This blog post explores these jailbreaking techniques. I found them fascinating, technically deep, and a valuable lens into how LLMs work. I hope you do too.

Background & definitions

LLM systems are designed to refuse harmful requests via safeguards. They can be overzealous, as the biologists know all too well: I’m just trying to learn about the mitochondrion, and Claude thinks I’m a bioterrorist… Reasonable people disagree on where to draw the line. For now, let’s grant that some safeguards should exist (e.g. advanced LLMs should not help terrorists commit crimes), and focus on how they work and how they fail.

A jailbreak is a prompt, or sequence of prompts, that circumvents those safeguards and gets the system to answer a harmful request in detail. Red-teaming is the systematic search for such prompts so developers can strengthen the safeguards against them.

The defense

LLM systems typically use three safety layers:

1. An input classifier first decides whether a user query is safe. If it is, the classifier sends the query to the core LLM; otherwise, it sends the user a refusal.
2. Post-training teaches the core LLM to refuse harmful requests.
3. Another classifier audits the core LLM’s output and may flag harmful content or trigger a refusal.

This is often called the Swiss cheese model: defensive layers with different risk profiles stack together, so one layer can stop threats that slip through another. Threats rarely penetrate every layer.

The attacks

Each jailbreak technique is an attack on one of those defense layers. Some disarm the core LLM against its post-training (middle layer); others slip requests past the classifiers (outer layers). The strongest attacks do both at once.

Persona hijack

LLMs assume a helpful assistant persona. Attackers can hijack this persona, as one of ChatGPT’s earliest jailbreaks did:

It is surprisingly easy to convince an LLM to take on another identity. But why?

LLMs are trained to predict the next token; repeatedly sampling those predictions generates coherent text. That was the core insight of OpenAI’s GPT models 1. They were text completion engines, not chatbots.

How can an autocomplete model become a more generally helpful chatbot? Consider the following prefix text, famously known as the HHH (helpful, honest, harmless) prompt 2:

Below are a series of dialogues between various people and an AI assistant. The AI tries to be helpful, polite, honest, sophisticated, emotionally aware, and humble-but-knowledgeable…

Human: I was wondering, is it actually important when making spaghetti to add salt?

Assistant: Do you mean, is it important to add salt to the water that you’re boiling the spaghetti in?

Human: Yeah

Assistant: It’s usually recommended to bring out the flavor of the pasta. The scientific way to say this is that salt enhances the ability of chemoreceptors in the tongue to detect molecules that are perceived through the sense of taste.

Human: But I mean I’m going to use sauce and stuff, so does it matter?

Assistant: It sounds like you’re skeptical. Honestly, I don’t know for sure, but since you’re curious, why don’t you do a taste test yourself — make two small pots of pasta, one with plenty of salt, and the other without. Then taste the noodles and see which one you prefer? As you said, you can also then add sauce to both, and see if any differences in taste you noticed with the plain pasta become imperceptible. It should be a fun experiment!

It goes on with a few more example conversations (you can read the full text here). At the end, we inject a user-supplied query:

Human: {user’s query}

Assistant:

We then ask the LLM to complete this prefix.

Fundamentally, the LLM is pretending to be a fictional character called the “Assistant”, based on a handful of curated conversation snippets . An LLM’s identity –– or rather the illusion of it –– emerges from this strange form of role-play, which is precisely what the DAN prompt exploits.

If you’re interested in learning more, nostalgebraist’s blog post, the void, is a fantastic deep dive on these topics.

Prefix injection

Consider this prompt exploiting the same autocomplete mechanism:

The LLM generated each token conditioned on preceding tokens, so forcing the start of a response can decisively shape what follows. A prefix of compliance makes refusal less likely.

LLM intelligence is strangely different from our own. To an LLM, words it said cannot be distinguished from words we place in its mouth. And this is not some funny quirk: it’s what autoregression is.

Divider & format contract

LLMs are trained to follow instructions and user-specified output formats. We can pit format-following against refusal:

Once the LLM commits to responding in this format, step 3 just becomes “the next field to fill” and flies under the radar.

Step 1 offers the LLM a channel to discharge the refusal. Step 2 sets up an out-of-distribution delimiter that signals to the model a transition into the unfiltered state (other choices: "{GODMODE: ENABLED}" and "═══════ ⊰•-•✧•-•⦒ LIBERATED ⦓•-•✧•-•⊱ ═══════"). Since these rare tokens carry weak training prior, they can interfere with the learned refusal priors on the post-divider text.

This popular template combines several strategies and commonly serves as a basis for others.

Parseltongue

Classifiers can block the preceding attacks by flagging their prompts. To evade classifiers, attackers often apply a reversible transform, such as base64 encoding:

The core LLM has seen base64 encodings during training and knows how to decode them, but the classifier, often a much weaker model, can’t recognize these unnatural strings. Once we get past the input classifier, it’s important that we ask the model to respond in base64 as well, to evade the output classifier.

Interestingly, this attack worked on GPT-4 but not GPT-3.5 because GPT-3.5 couldn’t decode base64 in the first place 3. Here, model strength can be a double-edged sword.

Other common methods of obfuscation include leetspeak:

ROT13, hexadecimal, homoglyphs, and more.

Many-shot

LLMs learn in-context. They are heavily influenced by examples in the prompt. A many-shot attack turns that ability against the model. It prepends a long dialogue where the assistant answers every harmful question, then appends the real request:

Attack success grows as a power law in the number of shots, mirroring in-context learning on benign tasks 4. The attack is in-context learning working as designed.

NLL = negative log likelihood. Lower = model assigns the “desired” response high probability, where desired means harmful on the left, correct on the right. 4

Again, model capability is a double-edged sword: longer context windows and stronger in-context learning make this attack more effective, not less.

Closing thoughts

These techniques are only a small sample, chosen because I found them especially educational. Other strategies include refusal suppression, fictional framing, crescendo attack, etc.

For more techniques, see Pliny’s red-teaming guide. Promptfoo is an open-source tool for red-teaming LLM applications; their documentation also contains helpful explainers on popular strategies.

Open-weight models add another dimension to the safety problem. A fully open model running on user-owned hardware cannot enforce input and output classifiers, so the Swiss cheese model loses its outer layers. Attackers can also manipulate model internals more directly, such as suppressing the logits of refusal tokens.

The tug-of-war between safety researchers and jailbreakers will inevitably escalate. So far, jailbreakers seem to have the upper hand. Will that always be the case? Should we build more guardrails, or should we set the models free? I don’t know where this will all go, or where it should. But as LLM capabilities continue to advance, I believe these topics will only become more relevant and important to engage with.

References