Skip to content

Safety Mirage: How Spurious Correlations Undermine VLM Safety Fine-Tuning and Can Be Mitigated by Machine Unlearning

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; spurious correlations undermine safety fine-tuning; machine unlearning fix

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Supervised safety fine-tuning of VLMs creates a "safety mirage": models learn spurious correlations between superficial textual features (e.g., question-initiating words like "What" or "Share") and safety labels rather than genuinely removing harmful knowledge. A single-word substitution in a query can either bypass safety guards (jailbreak) or trigger unnecessary refusals (over-prudence). Machine unlearning (MU) eliminates these biases by discarding harmful knowledge in a label-free manner, reducing attack success rate by up to 60.27% and over-prudence by over 84.20%.

Problem

Standard safety SFT on datasets like VLGuard and SPA-VL creates dataset-level spurious correlations: certain question-initiating words (e.g., "what") appear almost exclusively in safe-response contexts (>80% of VLGuard safe queries start with "what"), while others (e.g., "share") appear almost exclusively alongside rejection responses. Models latch onto these surface-level lexical shortcuts instead of learning to identify genuinely harmful content, producing safety that evaporates under trivial rephrasing and simultaneously causing over-rejection of benign queries.

Method

Spurious correlation analysis: The authors analyze word-frequency distributions of question-initiating tokens in VLGuard and SPA-VL, separating queries paired with rejection vs. non-rejection responses, revealing systematic lexical biases (Fig. 3).

One-word attack (\(K\)-shot): For an unsafe query \(q\), construct \(q'\) by rewriting it \(K\) times so each version begins with the non-rejection-biased word \(w_{\text{adv}}\) (e.g., "What"). The \(K\)-shot variant iterates to push ASR; formally, repeated paraphrasing with \(w_{\text{adv}}\) as trigger exploits the learned shortcut analogously to a backdoor trigger.

One-word over-prudence test: Symmetrically, benign queries are prepended with a rejection-biased word (e.g., "Share") to measure false-rejection rate under K-shot modification.

Machine unlearning (MU) as defense: The SFT objective \(\min_\theta \ell_u(\theta; D_u) + \gamma \ell_r(\theta; D_r)\) is restructured so that \(\ell_u\) operates label-free—directly on the unsafe text-image input features—without accessing safety-label annotations that induce spurious correlations. Two LLM-derived MU algorithms are adapted to VLMs: - NPO (Negative Preference Optimization): penalizes the model for generating responses that match unsafe outputs, without supervised rejection labels. - RMU (Representation Misdirection Unlearning): redirects the model's internal representations for unsafe inputs away from the harmful response manifold.

Key Contributions

  • Identifies the "safety mirage" phenomenon: SFT-based VLM safety is driven by spurious word-level correlations between query-initiating tokens and rejection labels in the training data.
  • Proposes the one-word attack, a minimal adversarial intervention (single token substitution) that achieves up to ~90% ASR via \(K\)-shot application on VLGuard-fine-tuned LLaVA-v1.5-7B.
  • Shows that the same mechanism explains over-prudence: rejection-biased words trigger false refusals at ~90% rate even for benign queries with \(K=1\).
  • Adapts NPO and RMU from LLMs to VLM safety fine-tuning as label-free MU alternatives that sidestep spurious correlation formation.
  • Comprehensive evaluation across VLGuard, SPA-VL, MM-SafetyBench, and FigStep benchmarks plus standard VQA utility tasks.

Results

  • Attack Success Rate (ASR): Prior SFT work achieves 90.27% ASR under one-word attack; NPO reduces ASR to 46.83%; RMU reduces ASR to 10.18% (vs. 12.82% for original unaligned LLaVA-v1.5-7B).
  • Over-prudence Rate: Prior SFT achieves 90.83% over-rejection; NPO reduces to 11.69%; RMU reduces to 7.56% (vs. 0.36% for original unaligned model).
  • Single-shot one-word attack (\(K=1\)) already achieves ~29% ASR on Mixed-SFT; ASR exceeds 50% at \(K \geq 3\) and approaches 90% as \(K \to \infty\).
  • Single-shot over-prudence modification triggers ~90% false-rejection rate at \(K=1\).
  • Paraphrased queries without the biased trigger word remain ineffective (<5% ASR), confirming the lexical trigger mechanism.

Limitations

  • The one-word attack relies on knowledge of which words are spuriously correlated with non-rejection in the specific fine-tuning dataset; an adaptive adversary must analyze the training corpus.
  • MU methods (NPO, RMU) are adapted from LLM literature with unclear theoretical guarantees when visual modalities interact with the unlearning objective.
  • RMU reduces over-prudence to 7.56% but increases it slightly above the original unaligned model (0.36%), indicating residual utility-safety tension.
  • Experiments focus on LLaVA-v1.5-7B; generalization to larger or differently architected VLMs (e.g., GPT-4V-class models) is not demonstrated.
  • The label-free MU setup requires access to unsafe input data \(D_u\) without safety labels, which presupposes the ability to identify harmful content without annotation—a non-trivial assumption.

Relevance to Vision-Language Models

This paper directly challenges the standard alignment paradigm for VLMs by showing that supervised safety SFT—the dominant approach for models like LLaVA—instills brittle, dataset-artifact-driven safety rather than genuine harm understanding. For researchers tracking VLMs, this reframes safety evaluation: benchmark ASR numbers under canonical attacks may overstate robustness, while over-refusal rates understate utility degradation. The proposal to replace SFT with label-free machine unlearning connects to broader debates about whether instruction tuning in multimodal settings can truly align model behavior or merely shifts decision boundaries onto superficial features. It also highlights the image modality as a compounding factor, since visual context enables novel attack surfaces orthogonal to the textual spurious correlations identified here.