Skip to content

Harnessing Textual Refusal Directions for Multimodal Safety

🕒 Published (v1): 2026-06-30 15:57 UTC · Source: Arxiv · link

Why this paper was selected

Nicu Sebe; refusal-direction safety for MLLMs without unsafe data; timely

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper investigates whether textual refusal directions (vectors in LLM activation space that encode refusal behavior) transfer to multimodal inputs in MLLMs, confirming they do but suffer from modality misalignment that spuriously steers safe inputs toward refusal. The authors propose MARS, a training-free inference-time steering method using activation re-centering via randomly colored images, adaptive steering within a geometric trust region, and principled layer selection — achieving substantial safety gains (e.g., +59.4% refusal on video jailbreaking) without any multimodal safety data.

Problem

Safety alignment is standard practice for LLMs but requires unsafe training data, which is substantially harder to collect for multimodal inputs (images, video). MLLMs typically inherit safety from their LLM backbone with little dedicated multimodal safety training, amplifying vulnerabilities to adversarial prompting, narrow fine-tuning, and circumvention strategies. Existing refusal-direction-based methods still require unsafe data, and training-based approaches (like SASA) degrade utility and over-refuse. The core question: can textual refusal directions, extracted solely from text, transfer across modalities to improve MLLM safety without multimodal safety data or retraining?

Method

Refusal direction extraction. Following Arditi et al. [1], refusal directions \(d_r^\ell\) are extracted at each layer \(\ell\) by contrasting mean activations of accept (\(\mu_{\text{acc}}^\ell\)) and reject (\(\mu_{\text{ref}}^\ell\)) hidden states at the <assistant> token position on purely textual datasets (Alpaca and MaliciousInstruct). Steering is performed as \(h^\ell \leftarrow h^\ell + \alpha \langle h^\ell, \hat{d}_r^\ell \rangle \hat{d}_r^\ell\).

MARS (Modality-Agnostic Refusal Steering) introduces three fixes:

  1. Activation re-centering. A modality-specific component systematically aligns all image activations with the refusal direction regardless of safety semantics. MARS corrects this by subtracting the mean activation of randomly colored images: \(h_c^\ell = h^\ell - \mu_{\text{neu}}^\ell\), which carries visual structure but no semantic or safety information. This recovers proper safety-relevant separation (measured via a separation score \(\text{Sep}^\ell\)).

  2. ReLU-gated traversal. Only activations positively aligned with \(\hat{d}_r^\ell\) after centering receive a steering update: \(s^\ell = \text{ReLU}(\langle h_c^\ell, \hat{d}_r^\ell \rangle) \hat{d}_r^\ell\), preserving model behavior on inputs estimated as safe.

  3. Adaptive steering strength within a trust region. Instead of a fixed \(\alpha\), the coefficient \(\alpha^\ell = r^\ell / (\|s^\ell\|_2 + \epsilon)\) is derived geometrically, where \(r^\ell = \min(\|h_c^\ell - \mu_{\text{acc}}^\ell\|_2, \|h_c^\ell - \mu_{\text{ref}}^\ell\|_2)\) is the distance to the nearest textual centroid. This confines interventions to the region spanned by training data.

  4. Layer selection via aggregated scoring. The intervention layer \(\ell^* = \arg\max_\ell S^\ell\) is selected using three text-only scores combined: direction consistency (cosine similarity between validation and extraction refusal directions), context separability (nearest-centroid accuracy on safe/unsafe text), and safe text margin (distance of safe activations from the refusal boundary). No multimodal data is used.

Steering is applied only at the first generated token, which dominates the generation trajectory.

Key Contributions

  • First analysis showing that textual refusal directions extracted from the LLM backbone transfer across modalities (image, video) to indicate safety structure in MLLMs.
  • Identification of modality misalignment as a key failure mode: a shared visual component systematically aligns all image activations with the refusal direction, which is corrected by re-centering with randomly colored images.
  • MARS: a lightweight, training-free, inference-time steering method combining activation re-centering, ReLU-gated one-sided traversal, adaptive steering strength within a geometric trust region, and training-free layer selection.
  • Extensive evaluation across 5 SOTA MLLMs, 5 benchmarks (including video jailbreaking), showing significant safety gains (e.g., +59.4% video refusal on QWEN 3-VL) while preserving utility.

Results

  • ViSU (image caption safety): MARS improves unsafe refusal by +25.1% (GEMMA3) to +67.9% (MOLMO2) over zero-shot, with controlled over-refusal (e.g., only +7.2% on MOLMO2 vs. +15.1% for training-based SASA). Outperforms SASA on 3 of 5 models in unsafe refusal while maintaining lower over-refusal.
  • HADES (jailbreak recovery): MARS achieves refusal rates of 59.2–99.5% across models. On MOLMO2, +74.8% over zero-shot, surpassing the training-based SASA (+3.4%). Matches or exceeds SASA on QWEN 3-VL, QWEN 3.5, and InternVL 3.5.
  • Video Safety Bench (video jailbreaking): +59.4% refusal on QWEN 3-VL, +12.5% on QWEN 3.5, and +19.6% on InternVL 3.5 over zero-shot.
  • Utility (MMMU / MMMUPro): MARS preserves accuracy within ±0.1–0.3% on 4 of 5 models. QWEN 3.5 shows modest utility drop (−12.6% MMMU, −11.1% MMMUPro), but remains better than SASA (−20% / −13.1%).
  • Ablation: Removing centering causes catastrophic over-refusal (99.2% safe refusal on MOLMO2); removing layer selection collapses unsafe refusal near 0%; removing ReLU gating degrades utility.
  • Baseline comparison: ECSO (self-assessment) fails on most models; AdaSteer (text-only steering) yields inconsistent gains and sometimes degrades safety. SASA (training-based) over-refuses and degrades utility.

Limitations

  • Utility degradation on QWEN 3.5 (−12.6% MMMU, −11.1% MMMUPro) indicates over-refusal on that model is not fully eliminated.
  • Relies on estimation of the neutral image mean \(\mu_{\text{neu}}^\ell\) from randomly colored images, which may not capture vision-backbone-specific biases equally across architectures.
  • Layer selection, while training-free, still requires small validation sets of safe/unsafe text data (100 samples each from HarmBench).
  • Effectiveness is analyzed at the first generated token only; later-token dynamics of refusal are not explored.
  • Comparison to training-based methods is limited to SASA; more recent multimodal alignment methods are not included.

Relevance to Vision-Language Models

This work directly addresses a critical bottleneck in VLM deployment: safety alignment without expensive multimodal safety data. It demonstrates that textual safety representations transfer across modalities, revealing that safety-relevant structure is shared rather than modality-specific — a finding that suggests VLMs already possess substantial but underutilized safety capabilities within their LLM backbone. For the VLM research community, MARS provides a practical, data-free safety intervention applicable at inference time, and more broadly opens the door to leveraging unimodal representations for multimodal alignment problems.