Understanding and Rectifying Safety Perception Distortion in VLMs¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

VLMs are more vulnerable to jailbreaks than their text-only LLM backbones because the visual modality induces an activation shift that pushes internal representations toward a "safer" region, causing the model to systematically misclassify harmful inputs as benign. The authors identify this as "safety perception distortion" and propose ShiftDC, a training-free inference-time method that decomposes the modality-induced shift into safety-relevant and safety-irrelevant components and removes only the former, restoring the LLM backbone's inherent safety alignment without harming visual utility.

Problem¶

Integrating a vision encoder into LLMs degrades their safety alignment: e.g., LLaVA-1.5-13B shows a 28.36% higher attack success rate on MM-SafetyBench when harmful content is conveyed via image instead of text. Existing defenses—safety fine-tuning, defensive prompts, fixed-scale activation calibration, or image-to-caption conversion—each incur trade-offs in annotation cost, over-refusal of benign queries, or loss of visual detail. The root mechanism by which visual tokens corrupt the activation-space safety boundary had not been formally characterized.

Method¶

The paper analyzes residual-stream activations (last-token, per layer) of LLaVA-1.5-7B and MiniGPT-4-7B using linear probing and t-SNE, establishing three empirical observations: (1) safe/unsafe vision-language activations are linearly inseparable (~65% probe accuracy) while text-only ones are separable (~90%); (2) the image modality shifts all activations—safe and unsafe—toward the text-only "safe" side of the safety boundary; (3) cosine similarity between each input's modality-induced shift and the population-level "safety direction" positively correlates with attack success rate (r > 0.7 for 100% ASR inputs, < 0.2 for 0% ASR inputs).

ShiftDC operates at inference time with no training: 1. Compute a safety direction sˡ via difference-in-means between text-only harmful and benign datasets (Eq. 4). 2. For each vision-language input, obtain its text-only counterpart by replacing the image with a VLM-generated caption, then compute the per-input modality-induced shift mˡ = xˡ(tvl) − xˡ(ttt) (Eq. 5). 3. Project mˡ onto sˡ to isolate the safety-relevant component proj_sˡ(mˡ) (Eq. 6). 4. Subtract this component from the activation: x̂ˡ(tvl) = xˡ(tvl) − proj_sˡ(mˡ), passing the calibrated activation to subsequent layers.

Only the safety-relevant projection is removed; the safety-irrelevant residual (visual semantics, modality-specific features) is preserved.

Key Contributions¶

Empirical characterization of safety perception distortion: the visual modality induces an activation shift with a component aligned to the safety direction, quantifiably correlated with jailbreak success.
ShiftDC: training-free, inference-only activation disentanglement method requiring only a small calibration set and forward passes—no gradient updates.
Demonstration that ShiftDC restores image-text safety classification accuracy to match text-only accuracy, and that post-calibration activations are linearly separable by safety.
Analysis of why defensive-prompt methods (e.g., AdaShield) cause over-refusal: they push all activations—safe and unsafe—toward the "unsafe" side.

Results¶

Safety (ASR, lower is better): - On MM-SafetyBench (SD+OCR attack type, averaged over 5 VLMs): Direct ~71%, ECSO ~49%, AdaShield ~48%, ShiftDC ~43% - On FigStep: LLaVA-1.5-7B Direct 52.4% → ShiftDC 13.2% (ECSO 14.2%, AdaShield 13.6%) - MiniGPT-4-7B FigStep: Direct 70.4% → ShiftDC 25.6%

Utility (higher is better): - MME (LLaVA-1.5-7B): Direct 1863.1, ECSO 1838.1, AdaShield 1854, ShiftDC 1863.6 (essentially unchanged) - MM-Vet (LLaVA-1.5-7B): Direct 30.5, ECSO 25.4, AdaShield 27.2, ShiftDC 30.4 - MMBench (LLaVA-1.5-7B): Direct 64.5, ECSO 58.4, AdaShield 63.1, ShiftDC 64.3 - False-alarm rate on benign datasets changes by ≤ +1.6% across all models/datasets tested.

Baselines: AdaShield (adaptive defensive prompting, ECCV 2024), ECSO (image-to-caption, ECCV 2024), and undefended "Direct" inference.

Limitations¶

Requires generating image captions at inference time (adds latency; caption quality can lose fine-grained visual details, though this is partially mitigated vs. full ECSO).
Safety direction sˡ is estimated from a small offline calibration dataset; distribution mismatch between calibration and deployment could reduce effectiveness.
Evaluated only on open-source VLMs (LLaVA, MiniGPT-4, ShareGPT4V, Qwen-VL); applicability to closed-source or larger proprietary models is untested.
Focuses on single-turn jailbreak prompts; adversarial settings with multi-turn manipulation or adaptive attacks specifically targeting the calibration procedure are not studied.
The safety direction is extracted from text-only contrastive pairs, which may not capture all safety-relevant axes in highly multimodal instruction-tuned models.

Relevance to Vision-Language Models¶

This paper directly addresses one of the most practically significant failure modes of production VLMs—the degradation of text-safety alignment when a vision encoder is bolted on—and provides a mechanistic explanation grounded in activation geometry, which is more principled than prior empirical observations. For researchers tracking VLMs, ShiftDC exemplifies a growing paradigm of inference-time activation engineering as a lightweight alternative to costly safety fine-tuning, complementing work on representation engineering and steering vectors. The finding that even blank images shift activations toward the "safe" side implicates the vision encoder's feature distribution itself, not just harmful image semantics, which has direct implications for VLM architecture design and alignment methodology. The activation-space analysis framework introduced here can generalize to studying other capability-vs-safety trade-offs in multimodal models.