Skip to content

Solve the Missing First Step: Can VLMs Standardize Raw Heterogeneous Medical Data?

🕒 Published (v1): 2026-07-06 05:52 UTC · Source: Arxiv · link

Why this paper was selected

VLMs for medical data standardization — tackles missing preprocessing step in clinical AI pipelines

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Existing medical AI benchmarks assume clean, curated inputs, skipping the upstream step of raw data standardization that real clinical deployment requires. This paper introduces MDS-Bench, a 1,939-sample benchmark evaluating whether VLMs can identify source files, convert heterogeneous medical images, align annotations, and produce structured image-text pairs from raw dataset folders. Even the best model (Gemini 3 Flash) achieves only 48.6% end-to-end success, exposing a critical bottleneck.

Problem

Medical VLM benchmarks (MultiMedQA, PMC-VQA, GMAI-MMBench, etc.) universally assume standardized, curated inputs, leaving the upstream standardization step unevaluated. In real clinical deployments, data arrives as heterogeneous files—DICOM volumes, NIfTI scans, segmentation masks, label tables, metadata records—with implicit relationships encoded in directory structures and filenames. No benchmark measured whether VLMs can recover usable samples from this raw, fragmented state.

Method

MDS-Bench is constructed by selecting 100 diverse medical imaging datasets, tracing each of 1,939 target samples back to their original raw sources, and manually verifying standardized ground-truth outputs. An agentic VLM receives a raw dataset folder and a target schema; it must execute a four-stage reasoning pipeline: 1. Source identification — locate relevant raw files and records; 2. Visual standardization — convert DICOM/NIfTI/TIFF/video/etc. into a common image format; 3. Schema-constrained annotation — infer task type, modality, anatomy, labels, masks, bounding boxes, and provenance; produce per-image JSON and dataset-level JSON; 4. Consistency check — ensure the image, per-image annotation, and dataset annotation are mutually grounded.

Evaluation uses an eleven-metric protocol organized into five groups: Structure (SV, SSC), Semantic (SC), Content (IC, IV, INR, CF), Metadata (MSC, MSJ), and Joint (SCJ, E2E). Source matching acts as a gate—field-level scores (SC, IC, IV, INR, CF) are averaged only over samples with non-zero source match to avoid rewarding outputs grounded in the wrong file. The strict End-to-End pass \(e_i\) requires a valid image-JSON pair, source match \(s_i = 1\), schema validity \(g_i \geq 0.85\), semantic correctness \(q_i \geq 0.5\), and content fidelity \(f_i \geq 0.5\):

\[e_i = \mathbf{1}\!\left[p_i \wedge s_i = 1 \wedge g_i \geq 0.85 \wedge q_i \geq 0.5 \wedge f_i \geq 0.5\right]\]

All metrics are computed per-dataset then averaged equally across 100 datasets to prevent large-dataset dominance.

Key Contributions

  • MDS-Bench: first benchmark targeting the upstream raw medical data standardization step; 1,939 samples across 100 datasets, covering classification (43), segmentation (51), and detection (6) tasks.
  • Eleven-metric evaluation protocol spanning structure, semantics, content fidelity, metadata consistency, and strict joint success, with source-matching as an evaluation gate.
  • Staged reasoning prompt design that decomposes the task into source grounding, visual conversion, annotation alignment, and consistency checking.
  • Empirical analysis of nine frontier agentic VLMs, identifying the structure-to-joint performance gap as the dominant failure mode.

Results

  • Gemini 3 Flash achieves the highest E2E at 48.6% and SCJ at 56.3%; Claude Opus 4.6 is second on SCJ (52.3%) but lower on E2E (30.5%).
  • Schema validity (SV) is consistently high across all models (80.0–88.2%), but SSC drops to 48.8–62.0%, revealing that syntactically valid JSON frequently encodes incorrect semantics.
  • E2E scores range from 21.5% (Grok 4.20) to 48.6% (Gemini 3 Flash)—more than half of samples fail at least one pipeline condition even for the best model.
  • Error attribution (Table 2): Gemini 3 Flash fails fewer than 50% on only 51 datasets for joint metrics; Claude Haiku 4.5 fails on 75 datasets.
  • The structure-joint gap (SV ≈ 80–88% vs. E2E ≈ 22–49%) is consistent across classification, segmentation, and detection subsets (Figure 5).

Limitations

  • Detection subset is small (6 datasets), limiting conclusions about object-detection-specific standardization challenges.
  • Ground truth relies on model-assisted drafting with human verification; errors in the verification step could affect benchmark reliability.
  • Evaluation is conducted in an agentic coding-agent environment; results may not transfer to non-agentic VLM deployment settings.
  • The benchmark reflects a fixed snapshot of 100 datasets; generalization to proprietary hospital formats (e.g., vendor-specific PACS exports) is untested.
  • Only nine frontier models evaluated; open-source or fine-tuned medical VLMs are not included.

Relevance to Vision-Language Models

MDS-Bench directly targets a blind spot in VLM evaluation: the assumption that multimodal inputs arrive pre-standardized. For researchers tracking VLMs, this work reframes capability assessment from perception-and-reasoning over clean data to the full pipeline including source grounding and format conversion—skills that matter for real-world deployment. The large structure-to-joint gap (schema validity ~85% vs. E2E ~35% on average) suggests that current VLMs overfit to schema compliance while failing at source-grounded semantic recovery, pointing to a concrete training and evaluation gap. The benchmark and eleven-metric protocol provide an infrastructure for future work on improving VLM robustness to heterogeneous, real-world multimodal inputs.