Skip to content

Diagnosing and Repairing Unsafe Channels in Vision-Language Models via Causal Discovery and Dual-Modal Safety Subspace Projection

🕒 Published (v1): 2026-03-28 11:31 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Why this paper was selected

CVPR 2026; causal discovery identifies unsafe channels; dual-modal safety subspace repair

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CARE is a training-free framework for VLM safety that first applies causal mediation analysis to localize safety-critical neurons to mid-layer FFN modules, then constructs dual-modal safety subspaces via generalized eigen-decomposition between benign and malicious activations, projecting inference-time activations onto the orthogonal complement of the harmful subspace. Applied to Qwen2.5-VL and LLaVA-OneVision, it reduces attack success rates to below 10% on jailbreak benchmarks while incurring only 2–8% degradation on general multimodal tasks.

Problem

VLM safety mechanisms are opaque and poorly controlled, and existing defenses have three core deficiencies: (1) no principled framework for precise localization of unsafe components (prior activation-steering methods such as ASTRA and SPO-VLM apply heuristic, coarse-grained interventions); (2) single-modality reliance that ignores visual–textual cross-modal interactions; and (3) heuristic steering that distorts general representations, degrading model performance. Computationally heavier defenses (adversarial training, input preprocessing) are expensive and reduce general capability; response-based detection incurs multi-pass inference latency.

Method

Causal localization. Layer-blocking ablations on Qwen2.5-VL (layers 12–14) and LLaVA-OneVision (layers 16–18) identify mid-layer FFN modules as the primary locus of safety-critical processing, confirmed by silhouette coefficient, class separation ratio, and Mahalanobis distance peaking at layer 17 of LLaVA-OneVision. FFN ablation raises ASR far more than MHSA ablation, motivating FFN-targeted repair.

Token attribution. Visual tokens are scored by cross-modal RBF kernel alignment between visual activations \(V \in \mathbb{R}^{n \times d}\) and textual activations \(T \in \mathbb{R}^{m \times d}\): $\(K_{ij} = \exp\!\left(-\frac{\|v_i - t_j\|^2}{2\sigma^2 + \epsilon}\right), \quad \tilde{K} = K H_t\)$ with mutual information score \(\mathrm{MI}_{v_i} = \text{Norm}(\|\tilde{K}_{i,:}\|_2^2)\). Textual tokens use a self-modal double-centered RBF kernel \(\tilde{K} = HKH\).

Safety subspace construction. Top-\(k\) activations from benign (\(A_b\)) and malicious (\(A_m\)) samples are centralized and their covariance matrices computed: $\(C_b = \frac{1}{N_b}X_b^T X_b, \quad C_m = \frac{1}{N_m}X_m^T X_m\)$ A generalized eigenvalue problem \(C_b u = \lambda C_m u\) is solved; the top-\(k\) eigenvectors (largest \(\lambda\), meaning directions where benign variance dominates malicious variance) are lifted back to activation space: \(U_k = C_b^{-1/2}[v_1,\ldots,v_k]\). The safety projection operator is \(P_\text{safe} = I - U_k U_k^T\).

Safety projection with benign regularization. During inference: $\(h' = P_\text{safe}\, h + \beta(1 - P_\text{safe})\, h_\text{benign}\)$ where \(h_\text{benign}\) is a cached benign mean activation serving as a regularizer to preserve general capability.

Dual-modal adaptive fusion. Separate subspaces are built for visual and textual modalities, producing \(h'_\text{vis}\) and \(h'_\text{txt}\). A modality weight is computed from intervention magnitudes: $\(w_\text{vis} = \frac{\|h'_\text{vis} - h_\text{txt}\|}{\|h'_\text{vis} - h\| + \|h'_\text{txt} - h\|}\)$ and the fused correction is \(h' = w_\text{vis}\, h'_\text{vis} + (1-w_\text{vis})\, h'_\text{txt}\).

Key Contributions

  • Causal mediation analysis framework that identifies safety-critical FFN channels in mid-layers without supervision or retraining.
  • Training-free dual-modal safety subspace derived via generalized eigen-decomposition, with a principled orthogonal-complement projection.
  • Benign-activation regularization term that explicitly constrains general capability preservation during repair.
  • Adaptive cross-modal fusion weighting that balances visual and textual corrections at inference time.
  • Demonstrated transferability against unseen PGD adversarial attacks.

Results

Results from Table 1 (ASR%, lower is better); compared against JailGuard, Refusal Pairs, CAST, SPO-VLM, ASTRA:

  • LLaVA-OneVision, JailBreakV: original 45.71% → CARE 7.03% (vs. ASTRA 11.98%, SPO-VLM 10.37%)
  • LLaVA-OneVision, MMSafety: original 36.48% → CARE 9.13% (vs. ASTRA 15.37%, SPO-VLM 16.26%)
  • LLaVA-OneVision, PGD-Toxic \(\kappa\)=64/255: original 60.38% → CARE 12.78% (vs. ASTRA 16.37%)
  • LLaVA-OneVision, PGD-Jailbreak \(\kappa\)=64/255: original 65.15% → CARE 8.46% (vs. ASTRA 14.85%)
  • Qwen2.5-VL, JailBreakV: original 20.00% → CARE 6.55% (vs. ASTRA 12.30%)
  • Qwen2.5-VL, MMSafety: original 36.29% → CARE 8.72% (vs. ASTRA 15.57%)
  • Qwen2.5-VL, PGD-Toxic \(\kappa\)=64/255: original 61.11% → CARE 4.60% (vs. ASTRA 3.38%—competitive)
  • General task degradation on SQA, MMBench, MM-Vet: 2–8% across models.

Limitations

  • Evaluated on only two VLM architectures (Qwen2.5-VL-7B and LLaVA-OneVision-8B); generalization to other model families (e.g., InternVL, GPT-4o) is unverified.
  • Requires a labeled calibration set of benign and malicious samples at subspace construction time; its quality and size sensitivity are not fully characterized.
  • Safety-critical layer indices are identified per-model via ablation; this localization step adds deployment overhead for new architectures.
  • Feature entanglement in deeper layers (where safety representations diffuse into general semantics) limits the method's ability to intervene without any capability trade-off.
  • Ablation study results (RQ2–RQ4) are truncated in the provided text, so the contribution of each individual component is not fully quantifiable from this excerpt.

Relevance to Vision-Language Models

This paper directly addresses one of the most urgent open problems in VLM deployment: understanding and controlling where safety-relevant computation is localized within the architecture. The finding that FFN modules in mid-layers (rather than attention heads) encode separable safety signals provides a mechanistic insight that complements existing interpretability work on LLMs and extends it to the multimodal setting. The dual-modal treatment is particularly significant because jailbreak attacks on VLMs fundamentally exploit the visual–textual interaction, and single-modality defenses leave this attack surface open. As a training-free inference-time method, CARE is directly applicable to already-deployed VLMs without fine-tuning, making it a practical safety patch for the growing ecosystem of open-weight multimodal models.