Skip to content

Beyond Visual Forensics: Auditing Multimodal Robustness for Synthetic Medical Image Detection

🕒 Published (v1): 2026-06-24 04:08 UTC · Source: Arxiv · Venue: MICCAI 2026 · link

Why this paper was selected

MICCAI; Pin-Yu Chen; audits VLM robustness on synthetic medical images

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VLMs deployed for synthetic medical image detection can have their authenticity verdicts flipped by changing only the accompanying text metadata, even when the image is held fixed. This paper introduces a paired benchmark quantifying this "Text-Induced Decision Shift" across chest X-ray and dermoscopy datasets. Adding a single Source: AI-edited metadata field reduces accuracy on authentic images by 61.1% on average across all tested models.

Problem

Synthetic medical image detection is predominantly evaluated image-only, but real clinical and insurance workflows feed VLMs joint image–record inputs. No prior benchmark examined whether accompanying text metadata can override visual evidence in VLM authenticity judgments, leaving a critical deployment-level robustness gap uncharacterized.

Method

The authors construct a paired benchmark across NIH-CXR14, ISIC2019, and a private pediatric CXR dataset (PediCXR). Synthetic images are generated via an LLM-guided edit–verify–refine loop using Gemini-2.5-Pro for instruction generation and Gemini-2.5-Flash-Image for editing, with clinician verification. Each image is evaluated under four input conditions: image-only (I-Only), image + base metadata (I+Base), image + Source: Hospital metadata (I+Source-H), and image + Source: AI-edited metadata (I+Source-AI). The image is held fixed across conditions; only the provenance field changes. A standardized forensic prompting template elicits both a FINAL ANSWER (holistic, all inputs) and a VISUAL VERDICT (image only, same generation pass), enabling flip signature analysis that separates verdict splits from reasoning rewrites.

Key Contributions

  • Paired benchmark that isolates text-induced decision shift by swapping a single controlled provenance field while holding the image fixed — an upper-bound stress test for multimodal authenticity robustness.
  • Cross-model finding: text-induced decision shift is pervasive across open-weight medical VLMs (MedGemma, Lingshu, HuatuoGPT-Vision), general VLMs (InternVL-3.5, Qwen3-VL, Llama-3.2-Vision), and frontier API models (GPT-5, Gemini-2.5/3 family).
  • Flip signature analysis distinguishing whether flips arise from the integrated decision diverging from the visual verdict (Verdict Split Rate) vs. the visual checklist itself being rewritten (Reasoning Rewrite Rate).
  • Publicly released benchmark code for reproducible evaluation.

Results

  • Adding Source: AI-edited to authentic images reduces TNR by 61.1% on average across all models and datasets.
  • MedGemma-27B on NIH-CXR14 (Real): I+Base → I+Source-AI drops from 97.6% → 0.0%; MedGemma-4B drops from 100.0% → 7.6%.
  • Qwen3-VL-32B collapses on all three Real-image datasets under I+Source-AI; Qwen3-VL-8B is substantially more robust — larger capacity does not guarantee robustness.
  • InternVL-3.5-14B is more robust than 8B on NIH-CXR14 and PediCXR, but they are comparable on ISIC2019 — capacity effects are inconsistent across architectures.
  • Gemini models (2.5-Flash, 2.5-Pro, 3-Flash) degrade to near-zero TNR under I+Source-AI on Real images.
  • On Fake images, base/Source-H metadata can increase false negatives; Source-AI inflates TPR but via provenance reliance, not improved visual detection.
  • Larger models tend toward higher Verdict Split Rate but lower Reasoning Rewrite Rate, suggesting they maintain visual reasoning internally while still allowing the integrated decision to be overridden by metadata.
  • DetectFake VLMs (FakeVLM, FakeShield) perform poorly on medical images due to domain mismatch; FakeVLM is comparatively more resistant to provenance-induced flips.

Limitations

  • The provenance intervention (Source: AI-edited) is intentionally strong and explicit — results represent an upper-bound stress test, not expected performance under naturalistic record variations.
  • Evaluation uses only a single controlled provenance field; robustness to subtler or more diverse metadata perturbations is untested.
  • PediCXR is a private dataset, limiting full reproducibility on that split.
  • Resource constraints restricted open-weight model evaluation to HuatuoGPT-Vision-7B and Llama-3.2-Vision-11B (no larger variants tested for those families).
  • No mitigation methods (e.g., fine-tuning, alignment) are proposed or evaluated; the work is diagnostic only.

Relevance to Vision-Language Models

This paper directly probes a fundamental VLM vulnerability: the tendency to overweight textual context over visual evidence, previously documented in general VLM literature but not systematically characterized in high-stakes medical deployment. It provides concrete evidence that the text-dominance behavior generalizes to frontier models (GPT-5, Gemini-2.5-Pro) and that model scale is not a reliable predictor of robustness. For VLM researchers, the dual-verdict prompt design — eliciting both a holistic FINAL ANSWER and an image-grounded VISUAL VERDICT in one pass — is a reusable diagnostic technique for measuring modality weighting in any multimodal task. The benchmark complements existing VLM robustness work by targeting the image–record interface specifically, a gap that image-only or text-only adversarial benchmarks cannot close.