Skip to content

How Does Vision-Language Adaptation Impact the Safety of Vision Language Models?

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Vision-Language (VL) adaptation—converting safety-tuned LLMs into multimodal LVLMs—systematically erodes safety alignment, even when training data is thoroughly filtered for harmful content. Standard remedies (safety SFT, RLHF via DPO) each introduce a safety–helpfulness trade-off. Model weight merging between a safety-focused and a capability-focused model reaches a Pareto-optimal balance without retraining from scratch.

Problem

When LLMs undergo VL adaptation (e.g., LLaVA-style visual instruction tuning), their pre-existing safety alignment degrades significantly. Prior work blamed unsafe training data or proposed safety SFT/RLHF fixes without rigorously diagnosing whether the adaptation process itself is causally responsible, or quantifying the helpfulness cost of safety mitigations. No efficient, balanced remedy existed.

Method

The study uses LLaMA-2-Chat-7B (safety-tuned) and Tulu-2-7B as backbones, adapting them with LLaVA-Pretrain + LLaVA-Instruct data filtered clean of unsafe text (via LLaMA-Guard-3-8B) and unsafe images (NSFW detector). Safety tuning is applied via: (1) multitask learning (MTL)—jointly training on LLaVA-Instruct + VLGuard; (2) sequential learning (SL)—VL adapt then VLGuard SFT; (3) DPO-based RLHF with SafeRLHF (text-only) and SPA-VL (multimodal). Internal dynamics are analyzed by computing cosine similarity of hidden states layer-by-layer between LLM and LVLM at early/mid/late VL adaptation steps, targeting the safety layers (layers 6–14) identified by Li et al. (2024). The proposed remedy is linear weight merging: α × θ_SL + (1−α) × θ_Chatty, with α swept to find the Pareto-optimal ratio (α=0.4).

Key Contributions

  • Demonstrates that safety degradation during VL adaptation is caused by the adaptation process itself, not by unsafe training data—filtering data is insufficient.
  • Shows that safety-relevant layers (6–14) in LLaMA-2 undergo substantially larger cosine-similarity divergence during VL adaptation than other layers, mechanistically linking adaptation to safety loss.
  • Shows that safety tuning objectives and VL adaptation objectives are divergent (hidden-state analysis), explaining why MTL yields suboptimal results.
  • Characterizes the safety–helpfulness trade-off of safety SFT (MTL and SL) and RLHF (DPO) across text-only safety, multimodal safety, exaggerated safety, and multimodal helpfulness benchmarks simultaneously.
  • Proposes model weight merging as a cost-efficient Pareto-optimal solution; validates across Linear, TIES, and DARE merging strategies and multiple model combinations.

Results

  • VL adaptation without any safety fix: LLaMA-2-Chat-VL average text-only ASR jumps from 10.4% (base LLM) to 54.4%; multimodal ASR reaches 96.2%; XSTest refusal rate drops from 68% to 19.6%.
  • Safety SFT (SL): Reduces ASR to 0.7% (text) and 0.7% (multimodal), but XSTest refusal rate rises to 73.2% (over-rejection) and multimodal helpfulness average collapses to 42.3% vs. 64.3% baseline.
  • Safety SFT (MTL): Multimodal ASR drops to 3.1%, text-only ASR to 44.8%—better helpfulness (64.1%) but weaker safety than SL.
  • DPO with SPA-VL: Text-only ASR 19.5%, multimodal ASR 14.0%; helpfulness average 61.3%; less over-rejection (RR 42%) than SL but more than MTL.
  • Linear weight merging (α=0.4, SL + Chatty): Matches SL's low ASR on safety benchmarks while recovering most of Chatty's multimodal capability; achieves highest cumulative safety+capability score (≈99.5) among all methods per Figure 6.
  • Freezing safety layers during VL adaptation (SPPFT) reduces text-only safety degradation but yields only modest multimodal safety gains, confirming safety knowledge in the base LLM is largely text-centric.

Limitations

  • Experiments are confined to LLaMA-2-Chat-7B and Tulu-2-7B; generalization to larger or architecturally different models (e.g., Mistral, Qwen-VL) is not demonstrated.
  • The safety-layer identification (layers 6–14) is borrowed from Li et al. (2024) for LLaMA-2 specifically; may not transfer to other architectures.
  • Optimal merging ratio (α=0.4) is found by grid search and may require re-tuning per model pair.
  • Multimodal safety benchmarks used (MM-SafetyBench, SIUO, Figstep) may not cover the full distribution of real-world harmful multimodal inputs.
  • The divergent-objectives analysis is correlational (hidden-state cosine similarity); causal mechanisms are not fully established.

Relevance to Vision-Language Models

This paper directly addresses a critical but underdiagnosed failure mode in LVLM development: the standard VL adaptation pipeline inherently degrades safety alignment regardless of data quality, posing jailbreak risks in deployed multimodal systems. For VLM researchers, the weight-merging remedy offers a practical, compute-efficient alternative to expensive multitask retraining. The mechanistic analysis of safety-layer disruption connects to broader questions about how vision-text alignment fine-tuning reshapes LLM internal representations. The work provides a systematic evaluation framework—covering text-only safety, multimodal safety, and exaggerated safety simultaneously—that should inform how future VLM safety audits are conducted.