Skip to content

V-CECE: Visual Counterfactual Explanations via Conceptual Edits

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

V-CECE is a training-free, black-box framework for generating visual counterfactual explanations (CEs) through semantically grounded, step-by-step conceptual edits. It uses WordNet-based bipartite matching to find provably optimal concept-level edits, then executes them via Stable Diffusion inpainting guided by GroundingDINO+SAM. A core finding is that LVLMs as classifiers align far better with human semantic reasoning than CNN/ViT classifiers.

Problem

Existing counterfactual image generation methods either produce dispersed, non-interpretable edits or assume (without verification) that classifiers operate at human-level semantic understanding. No prior work systematically measures the "explanatory gap"—the discrepancy between what edits a neural classifier requires to flip its label versus what a human judges as sufficient.

Method

V-CECE has two components:

Explanation component: Given source class L and target class L*, it extracts concept sets from each image using object annotations. Concept-level insertions (I), deletions (D), and substitutions (S) are costed via shortest-path distances on WordNet (Dijkstra). The minimal-cost edit set E is computed by solving a bipartite matching with the Hungarian algorithm (O(mn log n)), yielding provably optimal edits. Actionability is enforced by assigning infinite cost to infeasible edits.

Edit ordering (three strategies): - Local: An LVLM (Claude 3.5 Sonnet) receives the current image plus remaining edits and selects the next action iteratively. - Global: Edits are ranked by an importance score computed over all images in L (frequency of each edit type that flips class), applied in descending order. - Local-Global: Local (scene-aware) edit selection ordered by global importance scores.

Generative component: GroundingDINO+SAM produces masks for target concepts; Stable Diffusion v1.5 Inpainting executes each edit. The classifier C evaluates each intermediate image; generation stops on label flip or when E is exhausted. Claude 3.5 Sonnet provides commonsense spatial grounding (placement, background fill) for add/remove operations. No fine-tuning of any module is performed.

Key Contributions

  • First training-free, plug-and-play black-box CE framework with theoretical optimality guarantees on concept-level edits via bipartite matching on WordNet.
  • Three edit-ordering strategies (Local, Global, Local-Global) enabling controlled tradeoff between scene-awareness and classifier bias exploitation.
  • Systematic empirical measurement of the human–model explanatory gap: CNNs require ~3 more edits than humans judge necessary; LVLMs closely match human label-flip thresholds.
  • Demonstration that Claude 3.5/3.7 Sonnet as classifiers achieve near-perfect success rate (96–99.8%) with fewer edits and higher image quality than DenseNet, and outperform TIME (a trained black-box baseline) on image quality metrics.
  • Discovery that enabling Chain-of-Thought ("thinking") in Claude 3.7 degrades visual classification performance (requires more edits, lower SR), consistent with literature on CoT hurting visual tasks.

Results

BDD100K: - V-CECE with Claude 3.5 Sonnet (Local-Global): SR 98.10%, CMMD 0.7970, Avg.|E| 2.44 — outperforms trained black-box baseline TIME (SR 81.8%, no image quality advantage) without any training. - V-CECE with DenseNet (Global): SR 85.8%, CMMD 0.5489, Avg.|E| 5.37 — substantially worse image quality and more edits required. - Human survey: DenseNet requires average 5.22 edits vs. human judgment of 2.21; Claude 3.5 Sonnet requires 2.19 vs. human 1.33 — far closer alignment. - Visually correct images: DenseNet 59.7%, Claude 3.5 Sonnet 81.2%, Claude 3.7 Sonnet 80.0%. - Claude 3.7 with thinking: SR 97.73% but FID 73.51, Avg.|E| 4.07 — worse than without thinking (SR 99.76%, FID 68.17, Avg.|E| 3.45).

Visual Genome: - V-CECE (Local-Global, Claude 3.5 Sonnet): SR 98.87%, Avg.|E| 2.96 vs. prior semantic-CE methods [8,9] requiring ~12.8 edits (no image generation). - V-CECE with ResNet18 (Local-Global): SR 98.43%, Avg.|E| 2.56. - Global explanations: V-CECE returns 12 non-zero-importance edits for Claude 3.5 Sonnet on Bedroom class vs. 121 reported by [8] — significantly less noise.

Limitations

  • Human evaluation cohort is small in size and scope; results are framed as early insights, not statistically robust conclusions.
  • Relies on object-level annotations (from datasets) for concept set construction; not fully automatic for arbitrary images.
  • Stable Diffusion v1.5 Inpainting introduces generation artifacts at higher edit counts, compounding error measurement.
  • No evaluation of white-box generative models or analysis of mask/segmentation choice impact.
  • Framework not yet evaluated on medical or other high-stakes domains.
  • LVLM classification uses majority-voted self-consistency (7 runs) to handle hallucinations, adding inference cost.

Relevance to Vision-Language Models

V-CECE directly benchmarks proprietary LVLMs (Claude 3 Haiku, 3.5 Sonnet, 3.7 Sonnet with/without CoT) as image classifiers in a counterfactual XAI setting, providing a rigorous semantic alignment test that standard VQA benchmarks miss. The finding that LVLMs align closely with human concept-level reasoning while CNNs do not constitutes evidence for the semantic grounding hypothesis central to VLM research. The CoT degradation result (thinking-mode Claude 3.7 requiring more edits and producing worse image quality) is a reproducible, task-specific finding relevant to understanding when chain-of-thought reasoning hurts multimodal performance. For VLM practitioners, V-CECE offers a plug-and-play diagnostic tool to audit any black-box vision model's semantic coherence without retraining.