CHI-Bench: Can AI Agents Automate End-to-End, Long-Horizon, Policy-Rich Healthcare Workflows?¶

🕒 Published (v1): 2026-05-15 00:00 UTC · Source: HuggingFace · link

Why this paper was selected

CHI-Bench: end-to-end automation of policy-dense healthcare workflows via agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

χ-Bench is an in-situ evaluation benchmark for AI agents on end-to-end U.S. healthcare administrative workflows spanning prior authorization (PA), utilization management (UM), and care management (CM). It exposes a high-fidelity simulator of 20 healthcare apps via 87 MCP tools and a 1,279-document managed-care operations handbook skill, testing 30 agent harness/model configurations. The best configuration (Claude Code + Opus 4.6) resolves only 28.0% of tasks; no agent clears 20% under strict reliability, and a single-session marathon run collapses performance to 3.8%.

Problem¶

Existing healthcare and long-horizon agent benchmarks fail to jointly stress three critical capabilities: policy density (decisions grounded in a large, shifting library of medical, insurance, and operational rules), multi-role composition (a single task requires sequential role handoffs with irreversible commits), and multilateral interaction (intermediate steps are multi-turn dialogs such as peer-to-peer review or patient outreach). No prior benchmark combines all three with in-situ world-state verification, leaving a blind spot for realistic enterprise automation.

Method¶

χ-Bench instantiates each task as a quadruple (instruction, containerized χ-World environment, role-scoped tool surface, two-layer verifier) formalized as a hierarchical POMDP $\mathcal{M} = (S, A, O, P, Z, R, \rho_0; \mathcal{H})$, where the hierarchy $\mathcal{H} := (\mathcal{G}, \nu, \mathcal{W})$ encodes role-agent specifications $\{(G_i, u_i, K_i)\}$ with irreversible handoffs.

The simulator (χ-World Engine, ~115K lines of Python) models 20 healthcare apps via 151 REST APIs exposed as 87 MCP tools across 3 MCP servers, populated with ~5,000 chart activities for 50 patients. Agents are guided by the Managed-Care Operations Handbook skill: 1,279 markdown documents organized as a progressive-disclosure wiki (role sub-skills + shared medical-library + platform appendix), authored with clinicians from Johns Hopkins Medicine.

Tasks are constructed via a three-step pipeline: (1) Claude Opus 4.7 generates initial world states from a case-terminal state conditioned on the state graph and handbook; (2) human annotators execute each case end-to-end on the live UI to produce ground-truth trajectories; (3) at least one practicing healthcare worker and five authors review each case for clinical precision. A composite verifier combines deterministic contract checks (world store, event log, artifacts) with a strict-majority rubric LLM judge: $R = \text{DeterministicPass} \wedge \text{JudgePass}$.

Evaluation spans 30 harness/model configurations: three first-party CLI harnesses (Claude Code, OpenAI Codex, Gemini CLI) paired with closed-weight frontier models, and four open-source frameworks (OpenClaw, Hermes, OAI Agents SDK, DeepAgents) paired with five open-weight models via OpenRouter. Metrics: pass@1, pass@3, and passˆ3 (all 3 trials pass).

Key Contributions¶

χ-Bench: first benchmark combining long-horizon tool calls, explicit policy-dense retrieval, multi-role composition, irreversible world state, hidden multilateral interaction, and in-situ verification in a single healthcare task
χ-World Engine: high-fidelity simulator (20 apps, 87 MCP tools, 29-status case state machines with legal transitions, reviewer-independence constraints, FHIR-grade encounter linkage)
Managed-Care Operations Handbook Skill: 1,279-document skill library—the largest agent skill evaluated at operational scale to date
75 curated long-horizon tasks (human: avg 21 steps, max 40), filtered from 523 candidates, with multi-reviewer clinical validation and residual-PHI scan
30-configuration sweep across proprietary and open-source harnesses with ROI quadrant analysis and reliability degradation curves (pass@k vs. passˆk)
χ-Bench-Arena: end-to-end provider–payer evaluation where both sides are automated simultaneously

Results¶

Best overall: Claude Code + Opus 4.6 at 28.0% pass@1 (18.7% passˆ3); Sonnet 4.6 (26.2%), Opus 4.7 (24.4%), Codex + GPT-5.5 (20.9%)
No configuration clears 20% on passˆ3 (strict reliability); Opus 4.6 drops 28.0→18.7%, GPT-5.5 drops 20.9→9.3%
Marathon run (all tasks in a single session): best drops to 3.8%
Arena (end-to-end provider–payer): best prior-auth agents collapse to 0%
Domain-level leaders: Opus 4.6 for UM (41.3%), Opus 4.7 for CM (32.0%), Codex + GPT-5.5 for PA (29.3%)
Sweet-spot ROI: OAI Agents + GLM-5.1 (18.7% pass@1, $0.27/trial) anchors the Pareto frontier low-cost end
Overpriced quadrant: all Grok 4.3 cells, OpenClaw + Qwen 3.6 Max, Gemini 3.1 Pro
Open-source best: OAI Agents + GLM-5.1 and Hermes + Kimi K2.6 both at 18.7% pass@1; DeepAgents underperforms (best 11.1%)

Limitations¶

75 tasks only, filtered from 523 candidates—breadth is limited by annotation cost and clinical-review throughput
Tasks cover U.S. managed-care operations specifically; generalization to other healthcare systems or countries is unstated
The arena section is incomplete in the provided text; full provider–payer automation results are not fully reported
Skill retrieval is unbounded in theory but the 1,279-document handbook is still a fixed, curated proxy for real-world policy churn
Ground-truth trajectories are single-path; alternative valid workflows may be penalized by the deterministic verifier layer
Cost and step counts vary widely across harnesses (e.g., OpenClaw + Claude Opus 4.7: $11.48/trial vs. DeepAgents + DeepSeek V4 Pro: $0.21/trial), making fair cross-harness comparison difficult

Relevance to Harnesses / Meta-Harnesses¶

χ-Bench is the most direct empirical comparison of agent harnesses in a demanding, policy-dense, long-horizon setting published to date: it pits seven harness architectures (Claude Code, Codex CLI, Gemini CLI, OpenClaw, Hermes, OAI Agents SDK, DeepAgents) against identical tasks, making harness choice a first-class variable rather than an incidental implementation detail. The results show that proprietary first-party CLIs—particularly Claude Code—outperform open-source frameworks on the same model, implying that harness-level design decisions (context management, tool-call orchestration, session continuity) meaningfully cap achievable performance. The catastrophic marathon-run collapse (28.0%→3.8%) directly indicts harness inability to maintain coherent state across a long sequence of irreversible, multi-role handoffs—a core meta-harness design challenge. The ROI quadrant analysis further provides a principled cost-performance framework that meta-harness designers can use to select sub-harness configurations dynamically based on task budget.