Contamination Detection for VLMs Using Multi‑Modal Semantic Perturbations

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Multi-modal semantic perturbations detect VLM benchmark data contamination

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Existing contamination-detection methods designed for LLMs fail to reliably identify test-set leakage in VLMs because they ignore the visual modality. This paper proposes detecting contamination by generating semantically perturbed image–question pairs (same composition, changed correct answer) via GPT-4o captioning + Flux ControlNet, then flagging models whose accuracy drops on the perturbed variants. The method is the first to simultaneously satisfy practicality, reliability, and consistency requirements across all tested VLM families and fine-tuning regimes.

Problem

Public VLM benchmarks may have leaked into proprietary pretraining corpora, inflating reported performance. Prior contamination detectors (n-gram overlap, guided prompting, CircularEval, choice confusion) were designed for text-only LLMs and ignore visual features; when adapted to VLMs they fail to meet at least one of three core requirements—practicality (no clean-model baseline needed), reliability (works across fine-tuning strategies), and consistency (signal scales with contamination degree).

Method

Given a benchmark question–image pair, the pipeline: 1. Randomly reassigns the correct answer to a different option. 2. Prompts GPT-4o to produce a dense image caption conditioned on the original question, original answer, and the new target answer—ensuring salient visual elements driving the change are emphasized. 3. Passes the caption and Canny edge maps of the original image through Flux + ControlNet to generate a perturbed image that preserves global composition while encoding the new correct answer. 4. Filters generated pairs for unambiguous answerability (manual or automated). 5. Detects contamination by comparing aggregate accuracy on original vs. perturbed benchmark: a significant accuracy drop (positive \(\Delta_{\text{orig}} - \Delta_{\text{perturbed}}\)) flags memorization, since clean models with genuine reasoning should perform equally well or better on the easier perturbed variants.

Contamination degree is formalized as \(\text{deg}_D(x) = \sum_{d \in D} \mathbf{1}_{\{x=d\}} \times n\), simplified to epoch count \(n\) under full-dataset fine-tuning.

Key Contributions

• Novel multi-modal semantic perturbation detection framework that is the only method satisfying all three requirements (practicality, reliability, consistency) across all tested settings.
• First systematic study comparing VLM contamination detection methods under diverse fine-tuning regimes (standard, LoRA, full-parameter) and contamination degrees (\(n \in \{1,2,3\}\) epochs).
• Demonstration that existing LLM-centric detectors (n-gram, guided prompting, shared likelihood, CircularEval, choice confusion, multi-modal leakage) are unreliable for VLMs.
• Perturbed benchmark datasets and code released publicly.

Results

• Ours (MMStar, LLaVA-v1.5-7B): All 6 contaminated checkpoints detected (100% success); clean model accuracy rises on perturbed set (+31.51 pp), all contaminated variants drop (−8.29 to −16.16 pp).
• Ours (MMStar, Qwen2-VL-7B): All 6 contaminated checkpoints detected; clean model +16.16 pp, contaminated models −7.07 to −43.03 pp.
• CircularEval: Fails to detect LLaVA-v1.5-7B LoRA (epochs 2–3) and Qwen2-VL-7B standard fine-tuning (epoch 2); 0/6 success on MMStar_G for LLaVA.
• Choice Confusion: Completely fails to detect LLaVA-v1.5-7B across all strategies and epoch counts.
• Multi-modal Leakage: Requires a clean-model baseline; fails reliability check for LLaVA 3-epoch standard fine-tuning; breaks consistency for Qwen2-VL-7B LoRA epoch 3.
• Failure rate of perturbation pipeline: ~1.8% of RealWorldQA (8/440) and ~3.4% of MMStar (17/495) images deviate enough that contaminated models answer both original and perturbed correctly.
• Automated filtering (replacing manual review) produces comparable detection results (Table 9).
• NaturalBench real-world counterfactuals replicate findings: contaminated models drop up to 45.95 pp.

Limitations

• Requires questions with strict visual dependence; cannot detect contamination on benchmarks answerable from text alone.
• Currently relies on manual filtering to remove low-quality generated images, though the paper shows this can be automated.
• Perturbation quality is bounded by current diffusion model capabilities (poor text rendering, complex geometry).
• Only fine-tuning-stage contamination is studied; pretraining-stage leakage is computationally prohibitive to ablate.
• Detection operates at dataset level (aggregate accuracy drop); sample-level detection is proposed but not fully validated.
• Evaluated on two benchmarks (MMStar, RealWorldQA) and two model families (LLaVA-v1.5-7B, Qwen2-VL-7B); generalization to other architectures or closed-source models is not demonstrated.

Relevance to Vision-Language Models

This work directly addresses benchmark integrity in VLM evaluation—a foundational concern as VLMs are increasingly deployed and compared on standard leaderboards. The perturbation-based detection framework exploits VLMs' multi-modal nature rather than treating them as text decoders, setting a methodological precedent for VLM-specific evaluation design. For researchers tracking VLMs, it introduces a practical diagnostic tool applicable before trusting any reported benchmark number, and highlights that ControlNet-based image editing is mature enough to serve as a rigorous evaluation instrument. The paper also implicitly argues that contamination-robust benchmarks should require visual grounding—a design principle relevant to anyone building or selecting VLM evaluation suites.