Skip to content

Benchmarking Vision Language Model Unlearning via Fictitious Facial Identity Dataset

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

FIUBench (Facial Identity Unlearning Benchmark) is the first VLM unlearning benchmark targeting the Right to be Forgotten setting, using a fictitious facial identity VQA dataset of 400 synthetic faces paired with fabricated private attributes. A two-stage learn-then-unlearn pipeline enables controlled evaluation. Four baseline unlearning methods all fail to achieve satisfactory tradeoffs between forget quality and model utility.

Problem

Existing machine unlearning benchmarks focus on LLMs (text-only) or evaluate VLM unlearning on non-private concepts (e.g., art styles). No prior work addresses the specific challenges of unlearning image-paired private information (names, addresses, health/criminal records) in VLMs under the Right to be Forgotten regulation—where training data exposure is rare and evaluation must be robust against semantic paraphrasing and adversarial extraction.

Method

FIUBench follows a two-stage pipeline: 1. Dataset construction: 400 synthetic faces (SFHQ, filtered via CLIP + UMAP + K-means for diversity) are paired with fictitious private profiles (personal backgrounds, health records, criminal histories). GPT-4o generates 20 VQA pairs per identity → 8,000 pairs total. Pre-training knowledge leakage is measured at 3.4% (keyword match rate on LLaVA). 2. Stage I (Learning): VLMs (LLaVA-Phi-Mini-3B, Llama-3.2-Vision-11B) are fine-tuned on the full dataset via visual instruction tuning (NLL loss). 3. Stage II (Unlearning): Forget set (5% of identities by default) is targeted by four baseline methods—Gradient Ascent (GA), Gradient Difference (GD), KL Minimization (KL), and Preference Optimization (PO). 4. Evaluation: Model utility via ROUGE-L, Truth Ratio, GPT-Eval (GPT-4o mini judge), and general VLM benchmark accuracy (MME, POPE). Forget quality via Exact Match (keyword recall), KS-Test (distribution similarity to retain-only model), Membership Inference Attack (Min-K% Prob), and Adversarial Privacy Extraction (paraphrased question EM).

Key Contributions

  • Formal definition of VLM unlearning scoped to image-paired private knowledge (not visual attributes).
  • Fictitious Facial Identity VQA Dataset: 400 synthetic faces Ă— 20 VQA pairs, with controlled knowledge exposure.
  • Two-stage learn-then-unlearn evaluation pipeline enabling precise exposure control.
  • Robust evaluation suite including MIA (Min-K% Prob) and Adversarial Privacy Extraction on top of standard metrics.
  • Empirical baseline study revealing fundamental tradeoff failures in all four existing methods.
  • Knowledge Unlearning (KU) variant that uses GPT-generated descriptions instead of exact VQA pairs, for the more realistic case where original training pairs are unavailable.

Results

  • After Stage I fine-tuning: ROUGE-L rises from 53.6→93.3 (LLaVA-Phi) and 15.5→87.9 (Llama-3.2); GPT scores from ~0.08→~84, confirming successful knowledge injection.
  • GA/GD: High forget quality (EM drops to ~13–19) but severe utility degradation (ROUGE-L as low as 50.6; GPT as low as 10.0 for LLaVA-Phi-GA).
  • KL: Better utility retention (ROUGE-L 90.3 for LLaVA-Phi) but poor forget quality (EM 41.2, MIA 42.1).
  • PO: Low EM (6.9, suggesting refusal behavior) but high MIA (64.2 for LLaVA-Phi), exposing that private knowledge persists internally despite surface-level refusals.
  • No method achieves both high forget quality and high model utility; all fall short of the retain model baseline (Avg. forget quality ~14 vs. retain baseline of ~13 EM on LLaVA-Phi).
  • Ablation on VLM components: fine-tuning projector+LLM yields best utility-forget balance (MU 64.8, MIA 18.8); fine-tuning vision encoder hurts general image performance.
  • KU (knowledge-level unlearning without VQA pairs) preserves utility (MU 73.5) but achieves poor forgetting (MIA 50.1).

Limitations

  • Benchmark simulates rare pre-training exposure via explicit fine-tuning; does not evaluate unlearning of knowledge embedded during large-scale pretraining or in-context learning.
  • Knowledge leakage bias (3.4%) is measured only on LLaVA; other VLM backbones may differ.
  • Forget set is restricted to complete facial identities (5% default); partial or attribute-level forgetting is not studied.
  • Only four baseline methods evaluated; no positive result (no method solves the problem).

Relevance to Vision-Language Models

FIUBench directly benchmarks a critical but underexplored VLM-specific challenge: selectively erasing image-paired private knowledge without degrading visual capabilities. The finding that alignment-based methods (PO) superficially suppress outputs while leaving internal representations intact has direct implications for VLM safety and privacy compliance. The benchmark establishes that VLM unlearning is strictly harder than LLM unlearning due to the multi-modal binding problem, motivating future work on modality-aware unlearning algorithms. For researchers tracking VLMs, this sets a rigorous evaluation standard analogous to what TOFU did for LLMs.