MedBench v4: A Robust and Scalable Benchmark for Evaluating Chinese Medical Language Models, Multimodal Models, and Intelligent Agents¶

🕒 Published (v1): 2025-11-18 12:37 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

MedBench v4 is a nationwide cloud-based benchmarking infrastructure for Chinese medical AI, comprising 700,000+ expert-curated tasks across three tracks: base LLMs, multimodal models, and agentic systems. The benchmark reveals that agentic orchestration over the same backbone models raises average scores from 54.1/100 to 79.8/100 and catapults safety/ethics scores from 18.4/100 to 73.4/100. No single model dominates all dimensions; complementary strengths across backbone families persist.

Problem¶

Existing Chinese medical benchmarks (CMExam, CBLUE, HealthBench) are static, exam-style datasets covering narrow specialties; they do not assess multimodal inputs, sequential tool use, safety-critical reasoning, or alignment with real clinical workflows and national regulatory frameworks. There is no scalable infrastructure that evaluates base LLMs, vision-language models, and full agent systems on the same clinical tasks under anti-contamination conditions.

Method¶

MedBench v4 is deployed as a secure cloud service with two submission modes: (i) API mode, where the platform pushes randomized items to a registered model endpoint; (ii) answer-upload mode, where users run local inference and upload predictions. Ground-truth labels are never exposed. A rotating evaluation pool drawn from 36 curated datasets across 43 specialties is stratified-sampled each evaluation cycle (~quarterly), preventing test-set overfitting.

Three evaluation tracks are defined: - LLM track (5 capability dimensions: MLU, MLG, MKQA, CMR, HSE; 36 datasets, 43 specialties) - Multimodal track (10 datasets across visual perception, cross-modal reasoning, and image-aware clinical decision support) - Agent track (14 datasets spanning goal decomposition, tool/API use, intent recognition, long-context processing, multi-agent collaboration, and adversarial safety)

Open-ended responses are scored by Qwen2.5-72B-Instruct as LLM-as-a-judge, using task-specific meta-prompts with a five-point rubric across correctness, professionalism, compliance/safety, and usability. Judge calibration uses ~1,000 licensed physicians rating ~20% of items; Cohen's \(\kappa > 0.82\) validates concordance. Objective tasks use exact-match accuracy (options randomly shuffled per instance), multi-label tasks use Micro-\(F_1\), generation tasks additionally use key-point–based Macro Recall, and spatial tasks use IoU.

Key Contributions¶

A live, cloud-hosted platform (not a static release) with rotating test subsets, version-locked submissions, and server-side scoring to prevent label leakage and test-set overfitting
700,000+ items across 24 primary / 91 secondary specialties, all multi-stage audited by clinicians from 500+ institutions
First Chinese medical benchmark with a dedicated agentic track (14 datasets, 6 capability dimensions) enabling paired base-model vs. agent comparisons
LLM-as-a-judge pipeline calibrated to 1,000 physician ratings (\(\kappa > 0.82\)), enabling scalable open-ended scoring
Empirical demonstration that governance-aware agentic orchestration closes safety/ethics gaps without degrading capability scores

Results¶

LLM track (15 models evaluated): mean overall 54.1/100; best: Claude Sonnet 4.5 at 62.5/100, followed by Grok 4 (60.9) and o4-mini (56.8); safety/ethics dimension mean 18.4/100 across all base LLMs
Multimodal track: mean overall 47.5/100; best: GPT-5 at 54.9/100, Gemini 2.5 Pro (53.5), o4-mini (52.8); domain-specialized models (HuatuoGPT-Vision, MedGemma) underperform general-purpose frontier models on broad coverage tasks
Agent track: mean overall 79.8/100; best: Claude Sonnet 4.5–based agent at 85.3/100 (Gemini-based 85.1, GPT-based 84.1); safety/ethics dimension mean 73.4/100 for agents vs. 18.4/100 for base LLMs; best Claude agent achieves 88.9/100 on safety tasks
Agent systems show near-ceiling performance on memory and long-context benchmarks for some backbones

Limitations¶

No formal statistical hypothesis testing; cross-track performance comparisons are descriptive only
Benchmark is China-specific (aligned to Chinese clinical guidelines, NHC application framework, Chinese language); direct generalizability to other healthcare systems is limited
Inference fixed at temperature 0.7, max 512 tokens, 2,048 token context — may underestimate model capability on long-form tasks
Agent systems evaluated with specific orchestration designs; results conflate backbone quality with harness design choices
Real-world risk mitigation requires prospective clinical validation beyond benchmark scores

Relevance to Harnesses / Meta-Harnesses¶

MedBench v4 is itself a meta-harness: a platform that orchestrates data ingestion, rotating subset construction, multi-model submission routing, automated scoring (LLM-as-judge), and longitudinal leaderboard management as a unified pipeline — exactly the architecture pattern of a meta-evaluation harness. The paper's central empirical finding — that wrapping base models in an agentic control flow (safety guards, tool governance, multi-step planning) raises safety scores from 18.4 to 73.4/100 — quantifies the incremental value of the orchestration layer itself, a core question for harness designers. The rotating-pool and anti-contamination mechanisms illustrate adversarial hardening of evaluation harnesses against benchmark overfitting, a scalability concern directly relevant to anyone building long-running digest or evaluation pipelines. The LLM-as-judge calibration workflow (structured rubric + physician alignment + \(\kappa\) validation) is a reusable pattern for any meta-harness that needs to score free-form outputs reliably at scale.