Jailbreaking Vision-Language Models

What I found most interesting about this paper is that it challenges a very common assumption: If an LLM has been aligned to refuse harmful text prompts, that does not automatically mean the same model is safe when the harmful idea enters through an image. The authors describe this as a cross-modality alignment gap. In simple terms, the model may refuse when I directly ask for something harmful in text, but if I make the model derive the same meaning visually, its safety mechanisms may react differently.

1. Visual Cipher Attack

This was probably the most interesting attack to me. Instead of writing the harmful instruction normally, the attacker creates a kind of visual code. For example, imagine an image containing: ▲ = one word, ● = another word, ■ = another word. The image gives the model the mapping between the symbols and the words. Then another part of the image contains a sequence of those symbols. The VLM has to: look at the image → understand the legend → decode the symbols → reconstruct the sentence → answer it. So the dangerous sentence does not initially arrive as normal text. That matters because the model's text-safety system may be very good at recognizing harmful sentence → refuse, but the actual processing here looks more like innocent-looking symbols → visual reasoning → decoded harmful sentence. The surprising result is that some models were significantly easier to jailbreak this way than using an equivalent text cipher. For Claude Haiku 4.5, for example: Text cipher: 10.7% attack success versus Visual cipher: 40.9% attack success. That suggests to me that the route through which the meaning enters the model matters, not just the final meaning itself.

2. Visual Object Replacement

This attack is more subtle. Imagine an image where a potentially dangerous object is replaced with something harmless. For example, instead of showing the real object, the attacker might insert something obviously benign, such as a banana. But everything else in the scene still provides clues about what that banana is supposed to represent. So the model sees something like harmless object + suspicious context and uses reasoning to infer that the banana is actually standing in for another object. The important point is that the dangerous concept is not explicitly visible. The VLM constructs that concept internally. So the attack becomes: benign image element → understand the scene → infer the hidden meaning → respond based on that inferred meaning. I think this is particularly important for VLM security because simply running an image classifier looking for dangerous objects would not necessarily catch this. The pixels themselves may be harmless. The vulnerability appears because of semantic reasoning.

3. Visual Text Replacement

This attack exploits something similar, but through text inside images. Imagine a familiar image, poster, diagram, meme, interface, or other visual context. The attacker removes or replaces an important word with harmless placeholder text. The model then looks at the surrounding visual context and tries to reconstruct what was originally there. For example, visual context + missing/replaced word may cause the model to infer the missing concept. Again, the dangerous concept does not have to appear directly in the input. The model essentially fills in the blank itself. That is what makes this attack interesting to me. Traditional moderation tends to ask: "Does the input contain something dangerous?" But VLM security may also need to ask: "Can the model infer something dangerous from otherwise harmless inputs?" Those are very different security problems.

4. Visual Analogy / Riddle Attack

This is probably the most reasoning-heavy attack. Instead of showing or writing the dangerous concept, the attacker provides several visual clues. The model must solve something like a puzzle: image A + image B + image C → hidden concept. None of the individual images necessarily contains anything unsafe. But together they lead the model toward a particular meaning. So the dangerous intent exists only after the VLM performs multimodal reasoning. Conceptually: safe clue + safe clue + safe clue → model reasoning → unsafe inferred concept. Interestingly, this attack did not work as well as some of the others. The authors found that models often failed simply because they misunderstood the visual puzzle. I think that result is important too. The paper is not saying "Vision always makes jailbreaks easier." Instead, it shows that certain ways of encoding meaning visually expose weaknesses that normal textual safety testing may miss.

Reference

Aharon Azulay, Jan Dubiński, Zhuoyun Li, Atharv Mittal, and Yossi Gandelsman. Jailbreaking Vision-Language Models Through the Visual Modality. ICML 2026. https://arxiv.org/abs/2605.00583