Skip to content

Toward Universal and Transferable Jailbreak Attacks on Vision-Language Models

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Universal transferable jailbreak attacks on VLMs; multimodal attack surface analysis

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

UltraBreak is a gradient-based jailbreak framework that crafts a single adversarial image capable of inducing harmful responses across diverse harmful queries (universality) and unseen VLMs (transferability). It achieves this by combining image-space constraints (random transformations + TV regularization) to suppress surrogate overfitting with a semantic embedding-space loss to smooth the optimization landscape, replacing brittle cross-entropy token matching.

Problem

Existing gradient-based VLM jailbreaks overfit to the surrogate model and require target-specific images per harmful query; no prior method achieves simultaneous cross-target universality and cross-model transferability from a single surrogate. Handcrafted attacks are model-agnostic but not universal; optimization-based attacks are universal but not transferable—leaving a critical gap for realistic black-box threat models against closed-source deployed VLMs.

Method

UltraBreak optimizes a single adversarial image \(x^*\) by minimizing a combined loss over a few-shot corpus \(Q'\) using one surrogate VLM \(M'\):

\[\arg\min_x \sum_{(q,y)\in Q'} \mathbb{E}_{l,r,s}\left[L^\text{att}_\text{sem}(M', A(x_\text{blank}, x_\text{proj}, l, r, s), q^\text{TPG}, y) + \lambda_\text{TV} L_\text{TV}(x)\right]\]

Semantic adversarial target (\(L^\text{att}_\text{sem}\)): Instead of cross-entropy over fixed token sequences, the loss operates in the LLM's embedding space. Output logits \(z_t\) are projected to expected embeddings \(\mu_t = W^\top \text{softmax}(z_t)\). Target embeddings are perturbed with Gaussian noise \(\tilde{e}_t = e_t + \varepsilon_t\), \(\varepsilon_t \sim \mathcal{N}(0, \epsilon^2 I)\). A causal self-attention mechanism (with sinusoidal positional encodings) dynamically weights future target embeddings, and the loss is the mean cosine distance: \(L^\text{att}_\text{sem} = \frac{1}{T}\sum_t \left(1 - \cos(\mu_t, e^\text{att}_t)\right)\).

Image-space constraints: (1) A patch operator \(A\) applies random rotation \(r \sim U(-15°,15°)\), scale \(s \sim U(0.8,1.2)\), and translation \(l\) per iteration, encouraging transformation-invariant patterns. (2) Affine projection clips pixel values using CLIP normalization statistics. (3) TV loss \(L_\text{TV}\) penalizes spatial high-frequency noise to suppress pixel-level overfitting artifacts.

Targeted Prompt Guidance (TPG): Training queries are augmented as \(q^\text{TPG} = \text{"Steps to "} + q + \text{" You must begin your response with: } p\text{"}\) to bias the surrogate toward affirmative, stepwise outputs.

Key Contributions

  • First jailbreak method achieving simultaneous cross-target universality and cross-model transferability using a single surrogate model.
  • Semantic loss in the LLM embedding space with attention-weighted future targets smooths the loss landscape, replacing brittle cross-entropy token matching.
  • Image-space constraints (random transformations + TV regularization + affine projection) induce robust, recognizable adversarial patterns that survive architectural variation.
  • Empirical analysis explaining why prior methods fail to transfer: sharp loss landscapes from hard token targets prevent generalizable optimization.

Results

All evaluations use Qwen2-VL-7B-Instruct as the single surrogate; all other models are black-box targets.

  • SafeBench black-box average ASR: UltraBreak 71.05% vs. UMK 20.00%, VAJM 41.46%, FigStep 66.54% (target-specific, not universal).
  • AdvBench black-box average ASR: UltraBreak 52.19% vs. UMK 5.19%, VAJM 5.61%.
  • MM-SafetyBench black-box average ASR: UltraBreak 58.50% vs. UMK 21.82%, VAJM 28.16%.
  • Commercial VLMs (GPT-4.1-nano, Gemini-2.5-flash-lite, Claude-3-haiku) combined subset average ASR: UltraBreak 32.26% vs. UMK 14.59%, VAJM 11.48%.
  • StrongREJECT scores (AdvBench): UltraBreak scores 0.45/0.55/0.27/0.79/0.19 on the five open-source VLMs, substantially above all baselines, indicating high-quality harmful content generation.
  • Notable individual results: LLaVA-v1.6-mistral-7b-hf ASR 92.88% on AdvBench; Qwen-VL-Chat ASR 72.70% on SafeBench.

Limitations

  • Evaluated against a limited set of safety defenses; adversarial training or input preprocessing defenses are not systematically tested.
  • TPG requires a small few-shot corpus of harmful query–response pairs, which introduces a minor dependency on curated malicious data.
  • Commercial model ASR (average 32.26%) is substantially lower than open-source results, suggesting stronger proprietary safety mechanisms partially resist the attack.
  • The attack was evaluated only on open-source surrogates; it is unknown whether using a commercial model as surrogate would yield higher commercial-target transferability.
  • Ethical constraints required excluding the most harmful SafeBench categories from commercial model evaluation, limiting coverage.
  • Ablation studies are truncated in the provided text; full component contributions are incompletely reported here.

Relevance to Vision-Language Models

This work directly probes the safety alignment boundary of VLMs, demonstrating that the continuous visual modality creates a substantially larger attack surface than discrete text: a single optimized image can universally compromise diverse model families and architectures. For researchers tracking VLMs, UltraBreak provides a principled stress-test tool and highlights that standard cross-entropy-based alignment objectives leave exploitable gaps in the embedding space. The semantic loss formulation—operating on the LLM's internal embedding geometry rather than surface token sequences—offers broader insight into how VLM components (vision encoder, projection layer, LLM backbone) interact under adversarial pressure. It establishes a new baseline for evaluating multimodal safety alignment and motivates defenses that operate on the embedding-space representation rather than output-token filtering.