Skip to content

Approximate Domain Unlearning for Vision-Language Models

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper introduces Approximate Domain Unlearning (ADU), a new task requiring VLMs to forget recognition of images from specified visual domains (e.g., illustration, sketch) while preserving accuracy on retained domains (e.g., real photographs). The authors propose Domain Disentangling Loss (DDL) and an Instance-wise Prompt Generator (InstaPG) to overcome the core obstacle: domain distributions are heavily entangled in CLIP's latent space, making naive entropy-maximization unlearning strategies ineffective.

Problem

Prior approximate unlearning work on VLMs targets only class-level forgetting, which is insufficient for fine-grained real-world needs (e.g., an autonomous driving system must recognize real cars but ignore illustrated cars on roadside ads). Domain distributions are far more entangled than class distributions in pre-trained VLM feature spaces, so directly adapting class unlearning losses (minimize cross-entropy for retained domains, maximize entropy for forgotten domains) fails to achieve domain-level control.

Method

Built on CLIP (ViT-B/16) with deep visual prompt tuning (8 learnable tokens across 9 transformer layers):

  1. Domain Disentangling Loss (DDL): An auxiliary loss combining (a) a domain classification cross-entropy L_CE that trains a fully-connected domain classifier on top of image features, explicitly pushing domain distributions apart, and (b) Maximum Mean Discrepancy MMD² maximized (not minimized, unlike standard domain adaptation) to widen inter-domain divergence in RKHS. Total: L_domain = γ·L_CE − λ·MMD².

  2. Instance-wise Prompt Generator (InstaPG): Embedded in an intermediate transformer block; generates per-image vision prompts via cross-attention where learnable prompts are queries and image patch features are keys/values, capturing instance-level domain variation that a fixed prompt cannot model.

  3. Total objective: L_total = L_memorize + L_forget + L_domain, where L_memorize is cross-entropy on retained-domain samples and L_forget is uniform-label cross-entropy (entropy maximization) on forgotten-domain samples.

Key Contributions

  • Defines the novel ADU task: selectively unlearn domain-specific recognition in VLMs without class-level forgetting.
  • DDL that deliberately inverts the standard MMD domain-alignment objective to disentangle (rather than align) domain feature distributions.
  • InstaPG cross-attention mechanism for adaptive per-image domain-aware prompt generation.
  • Empirical demonstration that state-of-the-art CLIP fine-tuning (LP++, CLIPFit) and class unlearning methods (BBF) are fundamentally insufficient for ADU.

Results

  • ImageNet (2 domains, |D_forget|=1): Ours H=77.02 vs. best baseline CLIPFit H=71.31; For=96.73 vs. 96.43 (Baseline).
  • Office-Home (4 domains): Ours H=69.96/73.58/75.89 at |D_forget|=1/2/3 vs. Baseline 52.59/54.19/59.47; For improved from ~40% (CLIPFit) to >64%.
  • Mini DomainNet (4 domains): Ours H=75.56/77.03/81.78 vs. Baseline 62.07/61.97/68.82; For >70% across all settings vs. CLIPFit ≤39%.
  • Domain classification accuracy on Office-Home rises from 25.80% (zero-shot CLIP) to 79.43% with full method, vs. 31.06% without DDL+InstaPG, confirming effective disentanglement.
  • Ablation: DDL alone and InstaPG alone each improve H, but combining both yields best results on both datasets.
  • Method is stable across a wide range of γ and λ hyperparameters and scales better with more training shots than the Baseline.

Limitations

  • Requires domain labels for all training samples; unavailable in many real-world settings (though pseudo-labeling experiments in appendix show partial mitigation).
  • Integration of advanced domain estimation methods is left as future work.
  • Evaluation is on multi-domain image classification benchmarks only; applicability to other VLM tasks (generation, VQA) is not explored.
  • Few-shot regime (8 shots per domain) may not fully reflect large-scale deployment scenarios.

Relevance to Vision-Language Models

ADU directly targets a practical risk of CLIP-style VLMs: their strong domain generalization causes them to indiscriminately recognize objects across visual styles, creating safety and privacy hazards. The work reveals a structural property of VLM feature spaces—domain entanglement—that prior prompt-tuning and unlearning literature has not addressed. By inverting the MMD objective from domain adaptation to produce domain disentanglement, this paper introduces a qualitatively new direction for fine-grained VLM control that is complementary to class-level unlearning and domain generalization. Researchers tracking VLMs should note this as a first step toward selective knowledge removal at the domain granularity, with implications for privacy compliance, safety-critical deployment, and efficient specialization.