Skip to content

CF-VLM:CounterFactual Vision-Language Fine-tuning

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CF-VLM is a counterfactual fine-tuning framework that augments VLM training with minimally edited image-text pairs to push models beyond superficial statistical correlations toward causal reasoning. It introduces three complementary losses—cross-modal alignment, counterfactual scenario discrimination (Lcsd), and fine-grained causal discrimination (Lfcd)—applied over factual/counterfactual data generated via SDXL. Applied to CLIP and Qwen-VL, it outperforms SOTA on compositional reasoning benchmarks and marginally reduces hallucinations.

Problem

Existing VLMs (CLIP, BLIP, etc.) rely on coarse statistical co-occurrence rather than causal relationships between visual and linguistic content. Prior contrastive methods (triplet loss, NegCLIP, TripletCLIP) improve binary matching but ignore why pairs match—failing on fine-grained tasks where single-attribute changes (color, spatial relation, action outcome) flip semantic validity. Single-modality counterfactual approaches (text-only or image-only) likewise lack joint image-text counterfactual supervision.

Method

CF-VLM fine-tunes a pretrained VLM (CLIP ViT-B/32 or Qwen-VL 7B) on factual image-text anchors augmented with two counterfactual data structures:

  1. Complete counterfactual scenario pairs (Icfk, Tcfk): both image and text are jointly edited to form a semantically distinct but internally coherent alternative scene.
  2. Minimally edited counterfactual images Icf_editj: a single critical attribute or causal-relation edit is applied to the anchor image, paired with the original text as a hard negative.

Counterfactual texts are generated by Qwen2-72B-Instruct with 3-shot CoT; counterfactual images are synthesized by fine-tuned SDXL 1.0 Base + Refiner (40+15 steps).

Three losses are combined as LCF-VLM = α·Lalign + β·Lcsd + γ·Lfcd:

  • Lalign: symmetric InfoNCE over factual pairs, preserving pretrained alignment.
  • Lcsd (hinge): enforces S(Ia,Ta) > S(Icfk,Tcfk) + m1 for all K complete counterfactuals, stabilizing the anchor representation against plausible alternative realities.
  • Lfcd (hinge, hardest negative): enforces S(Ia,Ta) > max_j S(Icf_editj,Ta) + m2, sharpening sensitivity to minimal causal edits by focusing only on the most confusing counterfactual.

Training uses AdamW (lr 1e-5, cosine decay, 500 warmup steps), batch size 256 on a single A100 (bf16), with a 1:1 factual-to-counterfactual ratio for 200K steps on CC12M (8.6M pairs) or 90K steps on CC3M (2.6M pairs).

Key Contributions

  • Joint image-and-text counterfactual generation pipeline using prompted LLM + SDXL diffusion.
  • Lcsd loss that discriminates factual anchor from complete counterfactual scenarios (anchoring "parallel realities").
  • Lfcd loss that targets the hardest minimal-edit negative, training causal sensitivity to single-attribute perturbations.
  • Demonstration that CF-VLM transfers across architectures (CLIP ViT-B/32, Qwen-VL 7B, LLaVA-1.5) without architectural changes.
  • Incidental hallucination reduction (POPE, MME) without explicit hallucination-suppression objectives.

Results

Compositional reasoning (CLIP ViT-B/32, CC12M): - ConMe avg: 59.13% vs. TripletCLIP 56.53% (+2.6), COGT-CLIP 57.9% (+1.2), Standard FT 54.43% (+4.7) - ARO avg: 89.35% vs. COGT-CLIP 87.75%, Structure-CLIP 84.05% - VL-Checklist avg: 88.4% vs. COGT-CLIP 84.0%

Compositional reasoning (Qwen-VL 7B, CC12M): - ConMe avg: 87.57% vs. Standard FT 82.6% (+4.97), TextNeg FT 84.13% (+3.44) - ARO avg: 93.2% (VG-Rel: 91.8%, VG-Attr: 94.6%) - VL-Checklist avg: 90.57% vs. Standard FT 87.87%

Hallucination (POPE): CF-VLM improves accuracy, precision, recall, F1 by 0.5–1.2 pp over Standard FT on all three POPE settings (Random, Popular, Adversarial).

Hallucination (MME): Notable gains on Color (+2.0) and Position (+1.4) subtasks vs. Standard FT.

Generalization (CLIP, zero-shot): ImageNet-1k top-1 +8.83%, top-5 +19.54%; MSCOCO R@5 +13.80%, Flickr30k R@5 +26.58% over baseline.

Ablations: - Full CF-VLM (both image+text counterfactuals) > text-only (ConMe 55.9) > image-only (56.1) > non-causal (54.4). - All three losses needed: removing Lfcd alone drops ConMe avg from 59.1 to 57.9. - Performance plateaus at ~60–80% counterfactual ratio, indicating diminishing returns.

Limitations

  • Counterfactual image generation via SDXL is computationally expensive and effectively doubles training data size; quality depends on diffusion model fidelity.
  • Counterfactual text generation relies on a large 72B LLM (Qwen2-72B-Instruct), adding upstream inference cost.
  • Hallucination improvements are marginal (0.5–1.2 pp on POPE) and are a byproduct, not a design target; the method is not validated on dedicated hallucination-mitigation benchmarks like HallusionBench.
  • Evaluated only on classification and retrieval; VQA, visual reasoning chains, and grounding tasks are noted as future work.
  • The dynamic counterfactual generation (DCF) strategy is described at high level; systematic analysis of generation quality or failure modes is absent.
  • Hyperparameters α, β, Îł and margins m1, m2 are not ablated in the paper.

Relevance to Vision-Language Models

CF-VLM directly addresses a core weakness in contrastive VLMs: the gap between surface-level pattern matching and causal semantic grounding. By formalizing counterfactual supervision into separate discrimination objectives (scenario-level vs. minimal-edit), it provides a principled training signal that existing InfoNCE and triplet frameworks lack. The framework is architecture-agnostic—validated on both dual-encoder CLIP and autoregressive LLM-based VLMs—making it broadly applicable as a fine-tuning stage. Its connection to hallucination mitigation links counterfactual training to the reliability and factual grounding agenda that is central to deploying VLMs in real-world settings.