HealthAgentBench: A Unified Benchmark Suite of Realistic Agentic Healthcare Environments for Challenging Frontier AI Agents¶

🕒 Published (v1): 2026-06-30 06:09 UTC · Source: Arxiv · link

Why this paper was selected

Healthcare agent benchmark (54 envs); MS Research; rigorous domain evaluation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

HealthAgentBench is a benchmark suite of 54 agentic healthcare tasks across 7 categories, each packaged as a terminal-based Docker environment requiring autonomous multi-step interaction with real clinical data. Frontier agents perform poorly: the best agent (Codex GPT-5.5) achieves only ~42% task success rate, indicating the benchmark is far from saturation. The suite uniquely combines high interaction realism with broad coverage of clinical modalities and workflow stages.

Problem¶

Existing healthcare AI benchmarks are saturated, static, or narrow in scope—primarily testing isolated QA over short fixed contexts. They fail to evaluate agents that must autonomously explore raw, heterogeneous, large-scale clinical data (e.g., gigapixel pathology slides, 3D CT volumes, longitudinal EHR databases) through multi-step tool use. No prior benchmark unifies multi-modal, multi-workflow, executable-environment evaluation across the full patient journey.

Method¶

Each of the 54 tasks is packaged as a Docker container (via the harbor framework) providing a terminal environment. Agents receive minimal natural-language instructions and must autonomously explore /data/, install tools, query databases, and write outputs to /workspace/submission.json. Tasks are sourced either by converting existing benchmarks (e.g., EHRSHOT → EHR Event Modelling) into agent environments, or by constructing new tasks via rule-based injection into curated datasets (e.g., injecting physiological inconsistencies into MIMIC-IV for EHR Data Quality Auditing). Gold labels are mounted only in a separate verifier step (never agent-visible), agent-facing identifiers are opaqued, and web browsing is disabled to prevent label lookup. Evaluation uses binary success/failure with task-specific thresholds set at near-expert performance (e.g., tumor tile-F1 \(\geq 0.90\), AUROC \(\geq\) human-engineered count+GBM baseline). The primary metric is overall task success rate: the fraction of 54 tasks consistently solved across 3 attempts.

Key Contributions¶

HealthAgentBench: 54 agentic tasks across 7 categories (X-ray Report Correction, Pathology Tumor Area Selection, CT Abnormality Classification, Clinical Trial Matching, EHR Data Quality Auditing, EHR Event Modelling, EHR Format Conversion) spanning 5 modalities and 4 clinical workflow stages.
Principled task-creation workflow: four-stage pipeline (selection criteria → sourcing → design principles → post-creation checks) with anti-cheat guarantees and low chance-success rates (typically \(<10\%\)).
Challenging frontier-agent evaluation: benchmark is far from saturation; best agent reaches only ~42% success, with clear model-family and harness-dependent differences.
Nuanced capability analysis: identifies two major bottlenecks—medical imaging tasks and tasks requiring large search spaces with compositional reasoning.
Open release: benchmark code and environments released publicly.

Results¶

Codex GPT-5.5 (best agent): ~42% task success rate across 54 tasks (3 attempts each = 162 trials).
Copilot CLI + Opus-4.8: 36%; Copilot CLI + GPT-5.5: 35%.
Best native Claude Code agent (Opus-4.8): 32%.
Weakest agent (Codex GPT-5.4-mini): 16%.
Agent harness matters: GPT-5.5 drops from 42% (Codex) to 35% (Copilot CLI); Opus-4.8 improves from 32% (Claude Code) to 36% (Copilot CLI), likely due to Copilot CLI's multi-agent sub-delegation architecture.
Medical imaging is the hardest category, especially for Claude Code models; Codex GPT-5.5 shows emerging capability there.
EHR research/modeling pipelines are where frontier agents show most promise.
All agents struggle on tasks combining large search spaces with compositional reasoning.

Limitations¶

Only 54 tasks total; small per-category sample sizes (5–15 tasks each) may limit statistical power for fine-grained comparisons.
Tasks requiring data download at runtime with user credentials create reproducibility friction.
Evaluation is binary (success/failure), which may obscure partial progress; task-specific metrics are secondary.
Web browsing disabled by design, which may underestimate real-world agent capabilities that legitimately use external resources.
Benchmarks 10 agents from two model families (GPT-5 series, Claude series); coverage of open-source or other proprietary models is absent.
The paper text is truncated—full category-level breakdown and ablations are not available here.

Relevance to Agentic AI / LLM Agents¶

HealthAgentBench directly benchmarks frontier coding agents (Claude Code, Codex, Copilot CLI) in long-horizon, tool-use-heavy environments—making it a direct stress test of capabilities central to the agentic AI research agenda. Its finding that harness architecture (single-agent vs. multi-agent sub-delegation) significantly shifts performance underscores that agent scaffolding design is as important as the underlying model. The benchmark's design philosophy—terminal environments, minimalist instructions, anti-cheat construction, binary verifiable outcomes—parallels general-domain agent benchmarks (OSWorld, WebArena, GAIA) and demonstrates how that paradigm transfers to a high-stakes domain. For researchers tracking LLM agents, it provides a non-saturated, multi-modal evaluation surface where compositional reasoning over large environments remains the core open challenge.