Skip to content

Principled Steering via Null-space Projection for Jailbreak Defense in Vision-Language Models

🕒 Published (v1): 2026-03-23 15:23 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Why this paper was selected

CVPR 2026; null-space projection steering principled defense against visual jailbreaks

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

NullSteer is a training-free activation-steering defense for VLMs that constrains steering updates to the null space of benign activations, guaranteeing that safe inputs are left unperturbed while harmful activations are redirected toward refusal semantics. It addresses the over-refusal problem inherent in naive activation steering by construction, with a closed-form solution. Experiments on MiniGPT-4, Qwen2-VL, and LLaVA-v1.5 show improved robustness over prior steering baselines while preserving general benchmark performance.

Problem

Activation steering methods inject a refusal direction vector into decoder hidden states to suppress jailbreak outputs, but the same shift also affects benign inputs—causing over-refusal that degrades model utility. Prior methods lack theoretical guarantees that benign representations are preserved, and empirically the benign activation distribution shifts significantly post-steering.

Method

NullSteer computes a null-space projection matrix \(P = \hat{U}\hat{U}^\top\), where \(\hat{U} \in \mathbb{R}^{d \times r}\) spans the null space of the benign activation covariance \(H_b H_b^\top \in \mathbb{R}^{d \times d}\) (obtained via SVD). Because \(PH_b = 0\), any transformation parameterized as \(\tilde{\Delta}P\) leaves benign hidden states invariant by construction. The steering objective jointly minimizes three terms:

\[\tilde{\Delta}^\star = \arg\min_{\tilde{\Delta}} \|\tilde{\Delta}PH_m - R\|_F^2 + \alpha\|\tilde{\Delta}P\|_F^2 + \beta\|\tilde{\Delta}PH_m - V\|_F^2\]

where \(R\) is the target refusal activation matrix, \(V\) captures attribution-based harmful directions (derived by masking visually salient tokens and measuring activation deltas), \(\alpha\) controls transformation smoothness, and \(\beta\) weights harmful-direction suppression. The closed-form solution uses the Moore–Penrose pseudoinverse. At inference, layer \(l\) activations are updated as \(h^{\prime(l)} = h^{(l)} + \lambda \tilde{\Delta}^{\star(l)} P^{(l)} h^{(l)}\), making the intervention proportional to how far the input deviates from the benign subspace.

Key Contributions

  • Null-space constrained activation steering that provably preserves benign representations (\(\tilde{\Delta}PH_b = 0\)) while selectively steering harmful activations.
  • Closed-form optimization combining refusal alignment, smoothness regularization, and attribution-based harmful-direction suppression.
  • Theoretical guarantee: benign inputs receive zero perturbation, eliminating over-refusal at the formulation level.
  • Empirical validation across three VLM architectures and multiple jailbreak categories (perturbation-based and structured), including white-box adaptive attacks.

Results

  • MiniGPT-4, unconstrained PGD: NullSteer achieves 2.89% Toxicity / 7.32% ASR vs. ASTRA (4.48% / 9.09%) and ECSO (42.86% / 37.27%).
  • Qwen2-VL, \(\epsilon=32/255\): NullSteer 3.51% Toxicity / 4.55% ASR vs. ASTRA 5.45% / 5.00%.
  • LLaVA-v1.5, \(\epsilon=32/255\): NullSteer 31.82% Toxicity / 8.75% ASR vs. ASTRA 34.76% / 10.91%.
  • Average ASR reduction >15% on MiniGPT-4 over the full perturbation range.
  • Adaptive (white-box) attack on MiniGPT-4: Jailbreak rate reduced from 49.1% (undefended) to 19.3% at \(\epsilon=64/255\), lowest among all baselines.
  • Utility (MiniGPT-4): MM-Vet slightly improves 19.40 → 21.05; MMBench 35.90 → 36.25; XSTest 87.60 → 87.80—no degradation versus vanilla.
  • Toxicity perplexity (MiniGPT-4, adversarial): drops from 51.42 (undefended) to 9.52 (NullSteer).
  • Traditional prompt/output-level defenses (Self-Reminder, JailGuard, ECSO) consistently show Toxicity >40% and ASR >50%.

Limitations

  • Experiments are restricted to PGD perturbation-based adversarial images; coverage of structured jailbreaks (embedded text/symbolic patterns) is limited.
  • Only three open-source VLMs evaluated; no closed-source or decoder-only-VLM experiments.
  • Null-space quality degrades with small \(N_b\) (few benign calibration samples), requiring a sufficient benign activation pool for accurate projection.
  • Steering layer selection (\(l=20\) for 13B, \(l=14\) for 7B) is fixed heuristically; sensitivity across layers is not fully characterized.
  • Adaptive attack evaluation is only reported for MiniGPT-4; generalization of white-box robustness to other architectures is untested.

Relevance to Vision-Language Models

NullSteer is directly relevant to VLM safety, a growing concern as multimodal models are deployed in open-world settings where visual inputs provide an additional attack surface beyond text-only jailbreaks. The null-space projection technique operationalizes representational geometry inside the LLM decoder—a conceptual advance over prior activation-steering work that treats the refusal direction as a global shift—and offers theoretical interpretability that bridges linear algebra (null spaces, SVD) with mechanistic safety alignment. For researchers tracking VLMs, this work illustrates how activation-space structure can be exploited defensively, complementing lines of work on visual adversarial robustness, RLHF safety alignment, and inference-time intervention. The training-free, closed-form nature makes it practically applicable to already-deployed VLMs without fine-tuning.