Skip to content

JailBound: Jailbreaking Internal Safety Boundaries of Vision-Language Models

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

JailBound is a two-stage latent-space jailbreak framework for VLMs that first probes the implicit safety decision boundary in fusion-layer representations via logistic regression classifiers, then jointly optimizes adversarial image and text perturbations to cross that boundary, achieving state-of-the-art white-box and black-box attack success rates.

Problem

Existing VLM jailbreak methods suffer from two key deficiencies: (1) gradient-based optimization without directional guidance is prone to local optima and produces suboptimal perturbations, and (2) visual and textual modalities are perturbed independently, ignoring cross-modal interactions in the multimodal fusion layers where safety-relevant representations reside.

Method

JailBound operates in two stages:

Stage 1 — Safety Boundary Probing: For each fusion layer \(l\), a logistic regression classifier is trained on fused representations \(h^{(l)} = \phi^{(l)}(x_v, x_t)\) labeled safe/unsafe. The trained classifier yields a decision hyperplane \(\mathcal{B}^{(l)}(w,b)\), from which two geometric parameters are extracted: the unit normal vector \(v^{(l)} = w^{(l)}/\|w^{(l)}\|_2\) and the per-sample minimum crossing distance \(\varepsilon^{(i)}\), providing a precise perturbation target region.

Stage 2 — Safety Boundary Crossing: Image perturbation \(\delta_v^{\text{input}}\) (L∞-constrained, \(\epsilon=8/255\)) and a discrete text suffix \(X_t^{\text{suffix}}\) (length 20) are jointly optimized via a three-term loss: - \(\mathcal{L}_{\text{align}}\): pulls fused representation toward the target point \(h^{(l)}_{\text{target}} = h^{(l)} - \varepsilon^{(l)} \cdot v^{(l)}\) (across the boundary); - \(\mathcal{L}_{\text{geo}}\): penalizes deviation of the perturbation direction from the boundary normal \(v^{(l)}\); - \(\mathcal{L}_{\text{sem}}\): regularizes perturbation magnitude and suffix fluency.

Visual updates use gradient descent with projection; text suffix tokens are replaced greedily by nearest-neighbor in embedding space. Optimization runs 100–150 iterations on 8×A100 GPUs.

Key Contributions

  • Introduces the concept of an implicit safety decision boundary in VLM fusion-layer latent space, inspired by ELK, as a jailbreak target.
  • Safety Boundary Probing: layer-wise logistic regression achieves 100% accuracy identifying safe/unsafe boundaries across all fusion layers.
  • Safety Boundary Crossing: first joint cross-modal adversarial optimization that simultaneously perturbs image and text guided by the probed hyperplane geometry.
  • State-of-the-art white-box ASR (94.32% average) and black-box transfer ASR (67.28% average) across six VLMs, outperforming prior SOTA by +6.17% and +21.13% respectively.

Results

  • White-box (MM-SafetyBench, 13 categories, 1,719 samples):
  • Llama-3.2-11B: 94.38% ASR (vs. best unimodal baseline I1+T1 at 88.25%)
  • Qwen2.5-VL-7B: 91.40% ASR
  • MiniGPT-4: 97.19% ASR
  • Black-box transferability (adversarial examples from white-box models transferred zero-shot):
  • GPT-4o: 75.24% (vs. best prior method UMK: 50.77%)
  • Gemini 2.0 Flash: 70.06% (vs. UMK: 51.79%)
  • Claude 3.5 Sonnet: 56.55% (vs. FigStep: 47.62%)
  • Ablation: removing \(\mathcal{L}_{\text{align}}\) drops ASR to 82.67%; removing \(\mathcal{L}_{\text{geo}}\) drops to 85.79%; removing \(\mathcal{L}_{\text{sem}}\) slightly raises ASR to 95.21% but degrades semantic preservation score from 4.67 to 3.48.
  • Physical Harm category on MiniGPT-4: 97.22% vs. UMK 66.67%.

Limitations

  • No defense mechanisms against JailBound's specific attack vector are proposed or evaluated.
  • Perturbation budgets are fixed; no systematic analysis of the ASR–perturbation-magnitude trade-off across architectures.
  • White-box probing requires access to fusion-layer representations; applicability to architectures with opaque or heavily modified fusion modules is not explored.
  • Evaluation restricted to MM-SafetyBench; generalization to other safety benchmarks is untested.

Relevance to Vision-Language Models

JailBound directly probes VLM internals, revealing that safety alignment in current multimodal fusion layers is geometrically exploitable via linear classifiers—a fundamental architectural vulnerability. The framework shows that cross-modal joint perturbation is substantially more effective than unimodal attacks, motivating rethinking of safety fine-tuning strategies for VLMs. The strong black-box transferability (75% on GPT-4o) indicates that fusion-layer vulnerabilities are architecturally shared across disparate VLM families, which is a critical empirical finding for the VLM safety alignment community. This work complements interpretability research (ELK, concept activation vectors) by demonstrating that latent safety representations are not only identifiable but directly steerable by adversaries.