Beyond Classification Accuracy: Neural-MedBench and the Need for Deeper Reasoning Benchmarks¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; argues current medical VLM benchmarks mask true clinical reasoning gaps

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Neural-MedBench is a 120-case, 200-task neurology benchmark designed to expose the gap between classification-level benchmark performance and genuine clinical reasoning in VLMs. Evaluating 16 models reveals that even top performers (MedGemma-27B, Gemini 2.5-Pro) achieve at most 30% pass@1 on direct diagnosis tasks, far below senior physicians at 40%. Error analysis shows reasoning failure (51%) dominates over perceptual failure (27%), pointing to a fundamental cognitive bottleneck in current VLMs.

Problem¶

Standard medical VLM benchmarks (MedMNIST, MultiMedQA, MMLU-Pro) emphasize breadth—scale and classification accuracy—creating an evaluation illusion: models appear clinically capable while failing at integrative diagnostic reasoning. No compact, depth-oriented neurology benchmark existed to probe multi-modal synthesis, uncertainty resolution, and structured justification under realistic clinical conditions.

Method¶

The authors curate 120 authentic neurology cases from ADNI, OASIS, Radiopaedia, and peer-reviewed case reports (screened from 2,000+ candidates) via a funnel pipeline: multimodal completeness filtering → expert curation by two senior neurologists and one neuroradiologist → ground-truth annotation of structured diagnostic narratives → consensus review with trivial cases removed via baseline model filtering. Each case yields on average 1.67 tasks (200 total) across three families: Differential Diagnosis, Lesion Recognition, and Rationale Generation, stratified into three difficulty levels (Level 1: unambiguous single-modality; Level 2: conflicting cross-modal evidence; Level 3: multi-turn iterative refinement).

Evaluation uses a hybrid two-stage protocol: (1) an LLM-based grader with neurology-specific rubrics, validated against board-certified neurologists (Pearson \(r > 0.9\)); (2) BERTScore for semantic fidelity on free-form rationale. Metrics include pass@1, pass@5, BERTScore, and a five-category manual error taxonomy (Perceptual Failure, Reasoning Failure, Knowledge Gap, Grounding Error, Visual Hallucination). Human baselines from medical students (\(n=5\)) and senior physicians (\(n=5\)) anchor model scores.

Key Contributions¶

Two-Axis Evaluation Framework: proposes orthogonal Breadth (scale/diversity) and Depth (reasoning fidelity) evaluation axes, arguing that competent clinical AI requires both.
Neural-MedBench release: first neurology-focused, multimodal, reasoning-intensive benchmark (120 cases, 200 tasks, three task families, three difficulty tiers, multi-sequence MRI + EHR + clinical notes).
Empirical evidence of axis independence: state-of-the-art models with high Breadth scores fail sharply on Neural-MedBench, validating the uncorrelation hypothesis.
Systematic error analysis: identifies reasoning failure as the dominant bottleneck (51% of errors), not perception.
Human performance baseline: blinded clinician evaluation enabling direct human–model gap quantification.
Open leaderboard and extensible testbed at https://neuromedbench.github.io/.

Results¶

Best pass@1 on Direct Diagnosis: MedGemma-27B-it at 30.0% vs. Senior Physician at 40.0%; GPT-5 at 36.7% (highest among base VLMs).
Best pass@5 overall: Gemini 2.5-Pro at 50.0% (Direct Diagnosis); no model consistently exceeds 50%.
Complex Disease pass@1: Senior Physician 35.5% vs. MedGemma 18.3%, Lingshu 21.7%, GPT-5 28.3%.
Multi-round dialogue pass@1: Senior Physician 15.0%; GPT-5 19.5% (only model to approach/exceed physicians here).
Older medical-specialized models (RadFM, Med-Flamingo) score 0% pass@1 on Direct Diagnosis.
Error taxonomy (100 incorrect responses): Reasoning Failure 51%, Perceptual Failure 27%, remaining 22% distributed across Knowledge Gap, Grounding Error, Visual Hallucination.
LLM grader–neurologist agreement: Pearson \(r > 0.9\).
Generalist frontier models (Gemini 2.5-Pro) outperform medical-specialized models on pass@5 (differential breadth) while medical-specialized models lead on pass@1 (diagnostic precision).

Limitations¶

Small scale by design: 120 cases and 200 tasks limits statistical power and population diversity.
Single domain: restricted to neurology; generalizability to other clinical specialties is untested.
LLM grader dependency: the automated evaluator is GPT-4o-based; reliability relies on the one-time clinician calibration and may not transfer across all output styles.
Zero-shot evaluation only: no fine-tuned or few-shot conditions tested, leaving open whether reasoning gaps close with adaptation.
Human baseline sample size: only \(n=5\) per clinician group, limiting statistical confidence in the human anchor.
Input normalization: text lengths were normalized, which may not reflect realistic variable-length clinical records.

Relevance to Vision-Language Models¶

Neural-MedBench directly challenges the prevailing assumption that high performance on large-scale medical VLM benchmarks (MedMNIST, VQA-RAD, MultiMedQA) signals clinical readiness, providing quantitative evidence that VLM reasoning under multimodal uncertainty remains severely underdeveloped. The finding that 51% of errors are reasoning failures—not perceptual—redirects VLM research from improving image encoders or pretraining corpora toward enhancing integrative inference and self-correction (metacognition). The Two-Axis Framework offers a principled blueprint for future VLM benchmark design: depth-oriented, expert-curated stress tests should complement large-scale breadth evaluations rather than replace them. The anchoring bias observed in multi-turn dialogue tasks (VLMs failing to revise hypotheses under disconfirming evidence) identifies a concrete, mechanistically tractable failure mode for VLM reasoning research.