Skip to content

Which Concepts to Forget and How to Refuse? Decomposing Concepts for Continual Unlearning in Large Vision-Language Models

🕒 Published (v1): 2026-03-23 02:07 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Why this paper was selected

CVPR 2026; concept decomposition enables continual selective unlearning in LVLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CORE (COncept-aware REfuser) is a continual unlearning framework for large vision-language models (LVLMs) that decomposes forget targets into fine-grained visual attributes and textual intents, then routes refusal generation through a mixture of concept-specialized experts. It addresses the failure mode where sequential unlearning corrupts shared multimodal representations, causing the model to refuse irrelevant queries or over-refuse retain data.

Problem

Sequential machine unlearning in LVLMs distorts entangled visual-linguistic representations, creating spurious correlations between unrelated inputs and refusal behaviors. As deletion requests accumulate, prior refusal patterns are overwritten (causing contextually misaligned refusals on older forget tasks) and non-forget queries that superficially resemble forget categories are incorrectly refused. Existing methods relying on retain sets fail to prevent these representation distortions across sequential tasks.

Method

CORE operates in two stages per unlearning task \(t\):

1. Concept Recognition (Which concept is it?) For each forget category \(k \in K_t\), an LLM generates a set of textual descriptions of visual attributes and linguistic intents \(C_{q,k}\). Concept modules \(\{E^{q,k}\}\) produce cosine-similarity-supervised activations measuring alignment between input features \((x^t_{img}, x^t_{txt})\) and the accumulated concept set \(C_q^{1:t}\). A concept modulator \(M\) is trained via cross-entropy to classify forget categories, outputting per-category weights \(\{m_k\}\) that suppress irrelevant concept activations: $\(\bar{E}^t_{q,i} = \bigoplus_{k \in K^{1:t}} m_k \cdot E^{q,k}(x^t_{q,i})\)$ Past category knowledge is retained via prototype replay from earlier tasks.

2. Concept-Aware Refusal Generation (How to refuse?) A fixed pool of \(N_R = 20\) specialized connection modules (refusers \(\{V_j\}\)) transform visual features to steer refusal responses. A router \(R\) takes refined concept activations \((\bar{E}^t_{img}, \bar{E}^t_{txt})\) and computes mixing weights \(\{\alpha_j\}\). The refuser output \(\Delta P(x^t_{img}) = \sum_j \alpha_j V_j(x^t_{img})\) is added to the pretrained connection module output.

Routing is guided by a conceptual relevance score between the current task \(t\) and each previous task \(t'\): $\(r^{t'} = \sigma\!\left(\operatorname{sim}(\bar{E}^t_{img}, \bar{E}^{t'}_{img}) \cdot \operatorname{sim}(\bar{E}^t_{txt}, \bar{E}^{t'}_{txt})\right)\)$ A contrastive routing loss \(L_{ref}\) encourages reuse of refusers from high-relevance tasks and repurposing of underutilized ones for novel concepts.

At inference, the refuser contribution is scaled by \(\beta\) (the highest relevance score to any previously unlearned task), suppressing over-refusal on retain data: \(P(\bar{x}_{img}) + \beta \cdot \Delta P(\bar{x}_{img})\).

Key Contributions

  • CORE framework: concept-decomposition-based continual unlearning that grounds refusals in explicit visual attributes and textual intents rather than raw feature updates.
  • Concept modulator that disambiguates overlapping semantic regions across accumulated forget categories via learned reweighting.
  • Mixture-of-refusers with conceptual-relevance-guided routing, enabling refuser reuse across semantically similar tasks and adaptation of underutilized refusers for novel ones.
  • Inference-time refusal calibration via per-query relevance scoring to prevent over-refusal on retain data.

Results

All metrics reported on Vicuna-7B + ViT-g/14 + Q-Former backbone; baselines include EWC, LwF, GMM, EProj, SCRUB, MoEAdapter, and O3.

  • Specificity (general LVLM benchmark preservation): CORE Last = 96.54% vs. next-best MoEAdapter 94.46%; weakest baselines (GMM, SCRUB) drop to 62–63%.
  • Context-aware Refusal Rate (CRR) — Last: CORE = 90.67% vs. O3 73.03%, MoEAdapter 52.82%; weakest baselines 41–60%.
  • Refusal Gap \(\Delta RR\) (lower is better) — Last: CORE = 3.74 vs. O3 9.03, MoEAdapter 31.98; most baselines 26–37.
  • Answer Rate on retain data — Last: CORE = 88.02% vs. O3 81.76%, MoEAdapter 54.25%.
  • CORE achieves top or tied performance on BERT Score, CLIP Score, and ROUGE-L for both forget and retain data across Avg and Last settings.
  • Sequential step analysis (Figure 3) shows CORE maintains stable retain and benchmark performance while conventional methods degrade monotonically.

Limitations

  • Evaluated on a single LVLM family (MiniGPT-based with Vicuna-7B / LLaMA-2-7B); generalization to other LVLM architectures is not directly demonstrated in the main text.
  • The concept pool grows monotonically across tasks (\(|C_q^{1:t}|\) increases), which may become computationally costly or semantically saturated at scale.
  • The fixed refuser pool size \(N_R = 20\) creates a capacity ceiling; behavior when \(N_R\) is exhausted by dissimilar task sequences is not fully analyzed.
  • Requires an LLM at setup time to generate concept descriptions per forget category, introducing a dependency outside the LVLM itself.
  • Prototype replay from previous tasks requires storing feature prototypes, adding memory overhead.

Relevance to Vision-Language Models

This work directly targets a practical deployment problem for LVLMs: how to comply with sequential data-deletion requests (AI regulations, user rights) without catastrophic forgetting or collateral refusal of legitimate queries. The concept-decomposition approach offers an interpretable alternative to black-box gradient-based unlearning, grounding refusal behavior in semantically meaningful visual and textual primitives rather than raw parameter updates. It connects to the broader LVLM alignment and safety literature—particularly refusal training and RLHF—by showing that concept-level disentanglement can achieve precise, context-aware refusals that standard fine-tuning approaches cannot. The mixture-of-experts routing mechanism is also relevant to parameter-efficient LVLM adaptation research, extending MoE ideas to the selective forgetting domain.