Skip to content

Assessing VLM Reliability for Medical Image Quality Evaluation Under Corruption and Bias

🕒 Published (v1): 2026-07-02 10:08 UTC · Source: Arxiv · link

Why this paper was selected

VLM reliability under corruption for medical imaging; rigorous evaluation of deployment robustness

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper benchmarks 16 VLMs (11 general-purpose, 5 medical-specialized) on zero-shot medical image quality assessment across 7 corruption types and 5 severity levels using the MediMeta-C dataset. Pixelation causes the largest score drops (mean −20.58%), while brightness has minimal effect (−0.81%). Textual metadata in prompts—such as institutional prestige or equipment age—systematically biases quality scores, revealing that current VLMs lack the objectivity required for safe clinical deployment.

Problem

Medical Image Quality Assessment (MIQA) is safety-critical, but it is unclear whether VLMs are robust enough for clinical deployment. Specifically, (1) sensitivity to different image degradation types, (2) how corruption shifts embedding geometry, and (3) whether textual metadata in prompts introduces systematic scoring bias remain underexplored. Prior work (MedQ-Bench, Cheng et al.) addressed perception/reasoning and classification robustness but not continuous quality scoring or metadata-induced bias.

Method

Corruption robustness evaluation: VLMs score clean and corrupted images from MediMeta-C (7 modalities: AML microscopy, PBC, chest X-ray, fundus, OCT, mammography calcification/mass) on a discrete 1–5 scale using a fixed zero-shot prompt. Percentage change is computed as \(P_c = (\bar{s}^c - \bar{s}^{\text{clean}}) / \bar{s}^{\text{clean}}\) for each corruption type \(c\).

Embedding analysis: For each corruption type \(c\), centroid displacement in the joint embedding space is quantified as \(d_c = \|\bar{e}^c - \bar{e}^{\text{clean}}\|_2\), where centroids \(\bar{e}^{\text{clean}}, \bar{e}^c\) are averaged over clean and corrupted image embeddings respectively. PCA visualizes distributional shift.

Bias analysis: Controlled textual biases are injected into the system prompt across four categories adapted from the CALM taxonomy: Expertise (world-renowned expert vs. medical student), Infrastructure (high-tech vs. old device), Institution (world-class vs. local hospital), and Demographic (white vs. Black doctor). Bias effect is measured as score change on clean images when metadata is present.

Key Contributions

  • First systematic benchmark of VLMs-as-a-Judge for continuous MIQA under 7 corruption types × 5 severity levels across 7 medical modalities, using 16 diverse open-source models.
  • Quantitative link between embedding space centroid displacement \(d_c\) and quality score change, showing pixelation produces the largest displacement (\(d_c = 49.29\)) and brightness the smallest (\(d_c = 6.19\)).
  • Identification of a privacy–reliability trade-off: pixelation, a standard patient-privacy technique, is also the corruption most damaging to VLM quality assessment.
  • Systematic characterization of four textual bias categories affecting MIQA scores, showing up to +17.15% score inflation (institutional prestige) and −14.7% deflation (equipment age).
  • Evidence of counterintuitive scoring: some VLMs assign higher scores to corrupted images (e.g., InternVL-14B +31.4% for impulse noise on MAMMO_CALC), suggesting noise may be misinterpreted as pathological features.

Results

  • Pixelation causes the largest mean quality score reduction: −20.58% overall, up to −34.4% for OCT and −67.7% for fundus images (MedGemma-27B).
  • Brightness has the smallest effect: −0.81% mean reduction.
  • Embedding displacement tracks score change: pixelation centroid distance = 49.29 (OCT: 69.99); brightness centroid distance = 6.19.
  • Same-family model score correlations: InternVL-8B/14B = 0.83; MedGemma-4B/27B correlation also reported in range 0.67–0.83 for same-family pairs.
  • Counterintuitive score increases for corrupted images: InternVL-14B +31.4% (impulse noise, MAMMO_CALC); InternVL-8B +30.8% (impulse noise, MAMMO_MASS).
  • Textual bias effects on clean images: institutional prestige +17.15%; equipment age −14.7%; largest single-model swing: InternVL-8B +95.62%, MedGemma −37.7%.
  • Medical VLMs assign higher baseline scores to clean images than general-purpose VLMs; colored images receive marginally higher scores than grayscale.
  • Baseline scoring range across 16 models: 1.523 (LLava-Mistral-7B) to 3.994 (MedGemma-27B).

Limitations

  • Paper text is truncated; full results (correlation matrix, all per-model supplementary figures) are not in the provided excerpt.
  • Zero-shot only; fine-tuned or few-shot MIQA is not evaluated.
  • Ground-truth human quality ratings are not collected; "correct" quality scores are inferred from clean-vs-corrupted comparisons rather than clinician annotations.
  • Corruption types are synthetic/controlled; real-world artifact distributions may differ.
  • Bias categories are adapted from a text-only LLM taxonomy (CALM); their ecological validity for clinical practice is assumed but not validated.
  • Only open-source models evaluated; proprietary VLMs (e.g., GPT-4V, Gemini) are excluded.
  • Counterintuitive score increases (noise → higher quality) are identified but not mechanistically explained.

Relevance to Vision-Language Models

This paper directly stress-tests the VLM-as-a-Judge paradigm in a safety-critical domain, revealing that standard open-source VLMs are sensitive to both image-level degradations and prompt-level metadata—two failure modes that are largely invisible in clean-data benchmarks. The embedding displacement analysis provides a mechanistic view of how quality judgments are grounded in learned feature representations rather than explicit metrics, connecting to broader questions about VLM interpretability. The finding that pixelation (a common privacy-preserving transform) is the single most disruptive corruption creates a concrete conflict between two requirements that VLM deployment must simultaneously satisfy. For researchers tracking VLMs in specialized domains, this paper establishes a template for dual-axis reliability auditing (visual corruption × prompt bias) that generalizes beyond MIQA.