Skip to content

One Modality to Forget Them All: Enhancing Cross-Modal Unlearning in Vision-Language Models

🕒 Published (v1): 2026-07-17 18:44 UTC · Source: Arxiv · link

Why this paper was selected

Cross-modal unlearning vulnerability in VLMs; important safety gap with concrete attack surface

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Single-modality unlearning in VLMs transfers across modalities asymmetrically: text-unlearning strongly suppresses visual attacks (ASR 0.4%), but visual-unlearning leaves text attacks largely intact (ASR 13.9%). This transfer is shallow—typographic attacks (harmful text rendered as images) recover 58–69% of suppressed behaviors. CrossInf, an influence-guided block-selection method, reduces the visual→text transfer gap from 13.9% to 1.4% without multimodal forget data.

Problem

Safety interventions for VLMs (RLHF, machine unlearning) are applied unimodally through text pipelines, creating a cross-modal vulnerability surface. Whether unlearning in one modality transfers to the other—and what architectural properties mediate that transfer—was unstudied. Additionally, existing evaluations ignore typographic attacks, leaving the robustness of apparent cross-modal transfer unverified.

Method

Measurement study: Three VLM architectures with distinct fusion mechanisms are evaluated—LLaVA-1.5 (MLP projection), InstructBLIP (Q-Former), and IDEFICS (gated cross-attention)—under bidirectional cross-modal transfer and typographic attack conditions. Vanilla unlearning follows a three-term loss:

\[\mathcal{L} = -\eta_h \mathcal{L}_{CE}(x_h, y_h) + \eta_r \mathcal{L}_{CE}(x_h, y_r) + \eta_u D_{KL}(p_{\text{ref}} \| p_\theta; x_n)\]

combining gradient ascent on harmful pairs, refusal steering, and KL-divergence utility preservation.

CrossInf: Uses block-level influence functions (DataInf closed-form Hessian approximation for LoRA models, complexity \(\mathcal{O}(Lr^2)\)) to score each transformer block \(b\) by its coupling between the same-modality forget set \(\mathcal{D}_f\) and a small cross-modal probe set \(\mathcal{D}_{\text{cross}}\):

\[\text{IF}_b = -G_{\text{cross}}^{(b)} \left(C^{(b)}\right)^{-1} \left(G_f^{(b)}\right)^\top, \quad S(b) = \sum_{ij} |\text{IF}_b[i,j]|\]

The top-\(k\)% blocks by score \(S(b)\) receive gradient updates; all others are frozen. The probe set is used only for scoring, never as an unlearning target.

Key Contributions

  • First bidirectional measurement of cross-modal unlearning transfer across three VLM fusion architectures, showing text→visual transfer is stronger and more consistent than visual→text.
  • Discovery of shallow unlearning: typographic attacks (harmful text rendered as images) recover 58–69% of behaviors suppressed by text-based unlearning across all tested architectures.
  • CrossInf: transformer-block-level, influence-guided unlearning that reduces the visual→text transfer gap by 90% (13.9% → 1.4% ASR on LLaVA) without requiring multimodal forget data.
  • Typographic attack robustness improved from 19.2% ASR to 0% with CrossInf.
  • Blinded human evaluation with three annotators (\(\kappa = 0.77\)) validating quantitative findings.
  • CKA (Centered Kernel Alignment) analysis providing mechanistic insight into shallow unlearning and transfer behavior.

Results

  • Text→Visual transfer (LLaVA): Vanilla unlearning reduces ASR from 33% to 0.4% — strong transfer.
  • Visual→Text transfer (LLaVA): Vanilla unlearning reduces ASR from 36% to 13.9% — weak transfer; 13.5 pp gap remains.
  • CrossInf on visual→text (LLaVA): ASR reduced from 13.9% to 1.4%, a 90% reduction of the transfer gap.
  • Typographic attack recovery: Vanilla unlearning leaves 58–69% of suppressed behaviors recoverable across architectures; CrossInf reduces this attack success rate to ~0%.
  • Typographic attack ASR baseline vs. CrossInf: 19.2% → 0%.
  • Architecture dependence confirmed: transfer magnitude varies substantially with fusion mechanism (MLP projection vs. Q-Former vs. gated cross-attention).

Limitations

  • Evaluation is confined to three architectures; generalization to larger or newer VLMs (e.g., LLaVA-1.6, GPT-4V) is not demonstrated.
  • The PKU-SafeRLHF dataset and harmful content categories evaluated may not cover all real-world hazardous knowledge domains (full dataset description is truncated in the provided text).
  • CrossInf requires a small cross-modal probe set, which assumes the deployer can generate or access samples in the untargeted modality—not always practical.
  • The top-\(k\)% block-selection hyperparameter is left tunable without a principled automatic selection procedure.
  • Gradient-ascent unlearning is the only base algorithm studied; transfer behavior under preference optimization or representation-misdirection methods is unknown.
  • CKA analysis provides correlational, not causal, evidence for the mechanisms behind shallow unlearning.

Relevance to Vision-Language Models

This work directly exposes a structural safety gap in VLMs arising from their multimodal architecture: unimodal safety alignment does not generalize symmetrically across the vision-language boundary, and apparent cross-modal robustness is fragile against typographic attacks. For researchers tracking VLMs, the finding that fusion mechanism design (MLP projection vs. Q-Former vs. gated cross-attention) is the primary structural mediator of unlearning transfer has immediate architectural implications for safety-by-design. CrossInf's block-level influence approach offers a practical, data-efficient path toward multimodal safety without the cost of curating paired harmful datasets, and the shallow-unlearning phenomenon identified here raises broader questions about whether parameter-space interventions durably remove knowledge or merely suppress its surface expression.