Skip to content

Stochastic Meta-Unlearning: Bridging Language Backbone and Multimodal Unlearning

🕒 Published (v1): 2026-07-21 01:35 UTC · Source: Arxiv · link

Why this paper was selected

Sijia Liu + Tianlong Chen; first principled multimodal unlearning bridging language and vision components

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Stochastic Meta-Unlearning (SMU) addresses the failure mode where language-backbone unlearning in VLMs does not guarantee forgetting under multimodal (image-conditioned) inference. SMU is a bilevel meta-learning framework that trains a backbone initialization using VLM-level forget-retain feedback in the outer loop while keeping unlearning updates local to the language backbone. It achieves the best forget-retain trade-off over direct VLM-side unlearning baselines and generalizes to unseen forgetting targets.

Problem

Existing machine unlearning methods for VLMs either (1) optimize the fully composed VLM, risking utility degradation across unrelated capabilities, or (2) unlearn only the language backbone using text-only supervision, with no guarantee that forgetting persists after recomposition with visual modules. The authors demonstrate empirically that a backbone successfully "forgetting" a concept under text-only queries can still recover that concept when image information is provided at VLM inference — a component-system mismatch that neither approach resolves.

Method

SMU formulates VLM unlearning as a bilevel meta-optimization:

  • Inner loop: Starting from initialization \(\theta_T\), apply \(K\) gradient steps of text-side language-backbone unlearning using operator \(m\): $\(\theta_{T,k+1} = \theta_{T,k} - \alpha \nabla_{\theta_{T,k}} \mathcal{L}^m_{\text{in}}\!\left(f_{\theta_{T,k}}; B^{\text{text},z}_{f,\text{in}}, B^{\text{text}}_{r,\text{in}}\right)\)$
  • Outer loop: Recompose the updated backbone \(\theta_{T,K}\) with frozen visual modules \(\phi\) and evaluate the composed VLM \(F_{\theta_{T,K},\phi}\) on both multimodal forget and retain queries. The outer loss: $\(\mathcal{L}_{\text{outer}}(\theta_T; e) = \lambda^{\text{mm}}_f \mathcal{L}_f(F_{\theta_{T,K},\phi}; B^{\text{mm},z,c}_f) + \lambda^{\text{mm}}_r \mathcal{L}_r(F_{\theta_{T,K},\phi}; B^{\text{mm},c}_r) + \text{(text terms)}\)$
  • Meta-update: \(\theta_T \leftarrow \theta_T - \beta \frac{d}{d\theta_T} \mathcal{L}_{\text{outer}}\), backpropagating through the \(K\) inner steps so the initialization is shaped by VLM-level feedback, though \(\phi\) is never updated.

Stochastic generalization: Each meta-training episode samples an environment \(e = (z, m, c) \sim p(\mathcal{E})\), randomizing over forgetting target \(z\), unlearning operator \(m\), and query condition \(c\). This prevents overfitting to a fixed deletion scenario and promotes transfer to unseen targets and operators.

At meta-test time, only text-side backbone unlearning is performed from the learned initialization \(\theta_T^\star\); no VLM-level compute is needed.

Key Contributions

  • Identifies the component-system mismatch: text-side unlearning of the language backbone does not guarantee multimodal forgetting after recomposition with frozen visual modules.
  • Proposes SMU, a bilevel meta-learning framework that uses VLM-level forget-retain feedback to train a backbone initialization whose text-side unlearning steps generalize to multimodal forgetting.
  • Introduces stochastic task sampling over \((z, m, c)\) tuples during meta-training, enabling transfer to unseen forgetting targets and robustness to different meta-test unlearning operators.
  • Demonstrates that SMU outperforms three direct VLM-side baselines (GradDiff, NPO, RMU) on two VLMs and two multimodal meme benchmarks across both text-only and multimodal evaluation settings.

Results

Evaluated on Qwen2.5-VL-7B-Instruct and InternVL3-8B on Hateful Memes and COVID Memes:

  • SMU reduces average Forget accuracy by 10.52 points vs. the strongest per-metric baseline.
  • SMU improves average Retain accuracy by 20.10 points and Test accuracy by 17.01 points vs. the strongest per-metric baseline.
  • SMU aggregate: 9.16% Forget, 75.23% Retain, 68.45% Test accuracy (averaged over all models, datasets, and evaluation settings).
  • GradDiff achieves lower Forget in some settings (e.g., 0.87% on Hateful Memes/Qwen) but collapses Retain/Test to 31.16%/40.20%; on COVID Memes/Qwen (Text→Multi), all three metrics drop to 0.00%.
  • NPO and RMU preserve utility better than GradDiff but leave high Forget: e.g., 73.30% and 77.06% on Hateful Memes/InternVL (Text→Multi), vs. SMU's 18.20%.
  • SMU generalizes across meta-test forgetting targets (Fig. 4) and remains effective when the meta-test operator differs from the meta-training operator, e.g., trained with GradDiff but tested with RMU or NPO (Fig. 5).

Limitations

  • Experiments are restricted to two VLMs (Qwen2.5-VL-7B, InternVL3-8B) and two meme-domain datasets (Hateful Memes, COVID Memes); generality to other domains and architectures is unconfirmed.
  • Bilevel meta-training requires backpropagation through \(K\) inner unlearning steps and VLM-level forward passes in the outer loop, adding computational cost relative to single-level baselines.
  • The visual encoder and projection module are kept entirely frozen; whether SMU's design holds when visual components also require updating is not addressed.
  • Cluster-based forget-set construction is a specific and structured protocol; performance under more adversarial or fine-grained deletion requests is not evaluated.
  • Transferability across very different unlearning operators or larger target distribution shifts is only partially validated (three operators; fixed backbone architecture).

Relevance to Vision-Language Models

This work directly targets a structural weakness in VLM-specific machine unlearning: the modular composition of language backbone and visual encoder creates recovery pathways that text-only unlearning cannot close. SMU's bilevel design connects to the broader VLM literature by treating the frozen visual modules as an evaluation oracle rather than an optimization target, which is a principled way to exploit VLM modularity without disturbing visual-linguistic alignment. For researchers tracking VLMs, this paper provides both an important negative result — that unimodal unlearning does not transfer to multimodal inference — and a meta-learning recipe that could generalize to other post-hoc editing tasks (e.g., concept removal, safety alignment, continual forgetting). It complements benchmark-focused VLM unlearning work (CLEAR, MLLMU-Bench, FIUBench) by introducing a method-side solution to the cross-modal leakage problem.