Skip to content

Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMs

🕒 Published (v1): 2026-07-04 00:06 UTC · Source: Arxiv · link

Why this paper was selected

Counterfactual grounding + hard-negative contrastive training exposes visual shortcut problem in medical VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Medical VLMs achieve strong benchmark accuracy by exploiting textual shortcuts rather than visual evidence: image substitution costs only ≤6.5 pp on medical benchmarks versus 48–61 pp on general-domain tasks. This paper introduces a counterfactual evaluation framework with eight grounding metrics and CORAL, a LoRA fine-tune of Qwen2.5-VL-7B trained with a Contrastive Grounding Objective (CGO) that penalizes answer invariance under CLIP-retrieved hard-negative image swaps. CORAL achieves +6.7 pp accuracy and −8.0 pp Visual Hallucination Rate over the matched base, while RL fine-tuning variants fail to improve either metric.

Problem

Standard accuracy benchmarks cannot distinguish whether medical VLMs reason from visual evidence or from learned textual co-occurrence priors (e.g., question phrasing predicting the answer without reading the image). Existing saliency-based grounding methods require architectural access and measure attribution rather than behavioral reliance. No calibrated, benchmark-level diagnostic existed for auditing this shortcut behavior in medical VQA.

Method

Counterfactual evaluation framework. For each (model, sample) pair, six image conditions are constructed: real \(x^{\text{real}}_v\), blank (channel-mean gray) \(x^{\text{blank}}_v\), patch-shuffled (32×32 patches, fixed seed) \(x^{\text{shuffle}}_v\), image-absent \(x^{\emptyset}_v\), CLIP-retrieved hard-negative \(x^{\text{HN}}_v\), and corrupted \(x^{\text{cor}}_v\). Eight metrics are derived: Visual Reliance Score \(\text{VRS} = \text{Acc}^{\text{real}} - \text{Acc}^{\text{shuffle}}\), Blank Drop \(\text{BD} = \text{Acc}^{\text{real}} - \text{Acc}^{\text{blank}}\), Visual Benefit/Hallucination Rates (VBR/VHR), Image Sensitivity \(\text{IS}_{\text{pred}}\) (on extracted answers, not raw strings), Novel Visual Claim Rate (NVCR), Hallucinated Visual Reasoning Rate (HVRR), and Hard-Negative Gap \(\text{HN-GAP} = \text{Acc}^{\text{real}} - \text{Acc}^{\text{HN}}\).

CORAL + CGO. CORAL is a LoRA adapter (\(r{=}16\), \(\alpha{=}32\), dropout 0.05; targets \(\{q,k,v,o\}\)_proj) on frozen Qwen2.5-VL-7B-Instruct, trained for 2,000 steps on 4×H100s. The Contrastive Grounding Objective augments standard GRPO correctness reward:

\[r_{\text{CGO}} = r_{\text{acc}} + \lambda \cdot \mathbf{1}\!\left[\text{Ans}(\hat{a}(x_v, x_t)) \neq \text{Ans}(\hat{a}(x^-, x_t))\right]\]

where \(x^-\) is a hard-negative image (CLIP-ViT-L/14 cosine similarity to \(x_v\) maximized, subject to \(\text{label}(x^-) \neq y_i\)) and \(\lambda{=}0.5\). Hard negatives are mined via FAISS exact inner-product search over 5,778 pairs across PMC-VQA, PathVQA-closed, and SLAKE-closed (mean cosine similarity 0.967). The contrastive term rewards answer changes under the swap, making language-prior-only solutions high-loss at training time.

Key Contributions

  • A counterfactual evaluation framework with six image conditions and eight calibrated behavioral grounding metrics (VRS, BD, VBR, VHR, IS\(_{\text{pred}}\), NVCR, HVRR, HN-GAP), with paired bootstrap CIs and McNemar tests.
  • Quantification of the medical-VLM grounding gap: CLIP-nearest-neighbor substitution costs ≤6.5 pp on medical benchmarks vs. 48–61 pp on general-domain tasks across four 7B models and ten benchmarks.
  • CORAL: a 7B LoRA medical VLM trained with CGO, with released framework, training code, and model weights.
  • Empirical demonstration that accuracy-targeted RL (GRPO) sharpens language priors without improving grounding; the RL(image) checkpoint achieves its highest accuracy on no-image prompts.

Results

  • Macro accuracy (n=400, four medical benchmarks): CORAL vs. Qwen2.5-VL-7B base: +6.7 pp (95% CI [+0.75, +12.5] pp, \(P(\Delta>0)=0.988\)).
  • VHR reduction: CORAL −8.0 pp (95% CI [−11.75, −4.25] pp, \(P<0.001\)), a 57% relative reduction; 0.140 → 0.060.
  • Per-benchmark highlights: PMC-VQA +15 pp accuracy (CI excludes zero); VQA-RAD −14 pp VHR (CI excludes zero, \(P=1.00\)).
  • RL baselines: Neither RL(text) (−0.3 pp, n.s.) nor RL(image) (+2.3 pp, CI crosses zero) achieves significant accuracy or grounding improvement.
  • Grounding gap: On medical benchmarks, BD is near zero for all models; on general-domain tasks (ChartQA, VQAv2, GQA), BD is 0.08–0.13 for the same model family.
  • IS\(_{\text{pred}}\) caveat: CORAL has the highest extracted-answer consistency under shuffle (0.698 vs. base 0.518), indicating 70% of shuffled-image answers match the real-image answer — an unfavorable result for CORAL acknowledged by the authors.
  • VRS trend: CORAL 0.165 vs. base 0.128 (\(P(\Delta>0)=0.86\)), directional but 95% CI crosses zero.

Limitations

  • Evaluation uses n=100 fixed items per benchmark (n=400 paired total), underpowering secondary metrics (VRS, BD).
  • Train/eval benchmark overlap: PMC-VQA appears in both CORAL training data and evaluation, confounding the PMC-VQA accuracy effect.
  • IS\(_{\text{pred}}\) cannot distinguish true answer invariance from distinct reasoning paths converging on the same choice.
  • Comparison is restricted to Qwen2.5-VL and MedVLThinker-family 7B models; generalization to LLaVA-Med, BiomedGPT, HuatuoGPT-Vision untested.
  • CORAL emits direct answers rather than CoT (a consequence of PMC-VQA training format), so NVCR/HVRR reasoning-chain metrics are undefined for CORAL.
  • CGO does not directly optimize VRS or BD; the model remains only a partial corrective toward full visual reliance.

Relevance to Vision-Language Models

This paper directly addresses a known failure mode of VLMs — shortcut learning via language priors — in the high-stakes medical domain, extending the general VQA-CP line of work with a medical-specific, CLIP-based hard-negative contrastive objective. The counterfactual grounding framework and metric suite (VRS, VHR, HN-GAP) are reusable infrastructure for auditing any VLM for behavioral visual reliance, complementing saliency-based interpretability methods. The finding that RL fine-tuning on image-paired data can increase language-prior reliance (RL(image) achieves highest no-image accuracy) is a direct warning for practitioners using GRPO-style training on VLMs. CORAL's CGO recipe — rewarding answer changes under visually confusable hard negatives — is a transferable training intervention applicable to any VLM domain where textual shortcuts pose safety risks.