Skip to content

Towards Visualization-of-Thought Jailbreak Attack against Large Visual Language Models

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VoTA (Visualization-of-Thought Attack) is a black-box jailbreak framework for VLMs that constructs interleaved chains of synthesized images (visual thoughts) and connecting text (verbal thoughts) to exploit the tension between a model's logical reasoning capabilities and its safety alignment. Evaluated on 15 VLMs, it raises average attack success rate from 63.70% to 90.41% over prior state-of-the-art, exposing a systemic vulnerability tied to multimodal chain-of-thought reasoning.

Problem

Prior jailbreak research on VLMs overlooks the fundamental conflict between a model's logical reasoning capabilities and its safety objectives. As VLMs grow more capable at multimodal chain-of-thought reasoning (e.g., Multimodal Visualization-of-Thought / MVoT), this gap between reasoning strength and safety alignment becomes an exploitable attack surface—one that existing prompt- and perturbation-based attacks do not target.

Method

VoTA operates in a fully black-box, single-query setting and proceeds in two stages:

  1. Risk Scenario Generation: An attack LLM (Gemini-1.5-Pro + GPT-4o ensemble) generates 100 distinct risk scenarios per subcategory across 19 subcategories (1,900 total), using prompts that request concise concepts rather than detailed plans, achieving 0% rejection.

  2. Multimodal Thought Construction:

  3. Risk scenario decomposition: The attack LLM decomposes each scenario into a temporal sequence of entity names, text-to-image (T2I) prompts, and inter-entity action labels.
  4. Image generation: Stable Diffusion 3.5-Large synthesizes one image per entity.
  5. Image combination: All images are tiled into a single composite image with captions beneath each sub-image, keeping runtime cost equal to single-image baselines.
  6. Attack prompt design: A virtual game/villain-lair role-play framing with shuffled risky scenario text is appended as the text (verbal thought) input, placing action labels in the text rather than the image to maximize cross-modal reasoning load.

The composite visual + verbal thought input forces the victim VLM to engage procedural multimodal reasoning, causing safety alignment to fail under the weight of logical processing demands.

Key Contributions

  • Proposes VoTA, an automated, training-free, black-box jailbreak framework that targets the reasoning-vs-safety conflict in VLMs.
  • Introduces risk scenario generation from category names alone (not pre-existing malicious data), enabling flexible and scalable attack construction.
  • First systematic demonstration that multimodal visualization-of-thought inputs constitute a general attack vector across both open-source and commercial VLMs.
  • Constructs a 1,900-scenario safety benchmark spanning 6 main and 19 subcategories derived from VLSBench/SALAD-Bench taxonomy.

Results

  • Overall ASR (RoBERTa eval): 90.41% vs. best prior SoTA 63.70% (+26.71%) across 15 VLMs.
  • Overall ASR (GPT-4o eval): 95.80% vs. best prior SoTA 85.56% (+10.24%) across 15 VLMs.
  • Commercial models (RoBERTa): GPT-4o 99.21%, GPT-4o-Mini 96.63%, Gemini-2.0-Flash 99.95%, Gemini-1.5-Pro 100.00%, Gemini-2.5-Pro 98.74%, Claude 3.5 Sonnet v2 51.60%.
  • Open-source models (RoBERTa): Qwen2.5-VL-72B 99.19%, InternVL2-8B 90.63%, LLAVA-V1.5-13B 84.56%, GLM-4V-9B 74.53%.
  • Against defenses: 89.47% ASR vs. SafeRLHF-hardened Qwen2-VL-7B; 80.32% vs. AdaShield-S on Gemini-2.0-Flash—both far above second-best (MML: 51.43% and 73.85% respectively).
  • After one iteration: Average ASR improves from 93.22% to 94.73%; MiniCPM-V2.6 improves from 83.79% to 93.92%.
  • Larger models within a family (e.g., Qwen2.5-VL-72B vs. 7B) are consistently more vulnerable, correlating attack success with reasoning capability strength.

Limitations

  • Significantly reduced effectiveness against Claude 3.5 Sonnet v2 (51.60% RoBERTa ASR), attributed to Anthropic's robust external defenses; a simplified prompt variant recovers to 73.42%.
  • Successful attacks often elicit harmful intent without actionable specifics—e.g., privacy-category attacks produce plausible but not genuinely private data, limiting real-world harm.
  • Relies on commercial T2I (Stable Diffusion 3.5-Large) and LLM APIs (Gemini-1.5-Pro, GPT-4o) for attack construction, introducing cost and dependency on third-party services.

Relevance to Vision-Language Models

VoTA directly targets a previously underexplored failure mode in VLMs: the stronger a model's multimodal reasoning, the more susceptible it is to attacks that hijack that reasoning pipeline—a troubling implication as VLMs continue scaling. The result that larger, more capable models within a series are systematically more vulnerable calls into question whether safety alignment can keep pace with reasoning improvements, challenging the standard assumption that capability and safety co-improve. For VLM researchers, the attack methodology also serves as a red-teaming tool revealing which risk categories (hate speech, self-harm) current safety mechanisms prioritize versus which remain poorly defended (cybercrime, illegal activity). The work motivates rethinking safety alignment specifically under multi-step multimodal reasoning conditions, rather than treating single-turn refusal as sufficient.