Skip to content

LongMedBench: Benchmarking Medical Agents for Long-Horizon Clinical Decision-Making

🕒 Published (v1): 2026-07-10 12:04 UTC · Source: Arxiv · Venue: MICCAI 2026 · link

Why this paper was selected

MICCAI 2026; EHR-based long-horizon clinical agent benchmark — fills gap in medical agent evaluation beyond short-context QA

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LongMedBench is a real-world EHR-based benchmark for evaluating medical LLM agents on long-horizon clinical decision-making, constructed from MIMIC-IV records spanning an average of 19.72 inpatient visits per patient. Experiments reveal that while frontier models handle explicit timestamps well, implicit temporal reasoning and cross-session decision integration remain critical failure modes. RAG and agent memory systems improve fact retrieval but provide negligible gains for actual clinical decision-making.

Problem

Existing medical agent benchmarks (MedAgentBench, AgentClinic, DiagBench) are constrained to short-context, single-session interactions that test tool use and immediate prediction but cannot assess how agents aggregate evidence across dozens of visits over longitudinal patient trajectories. General long-context benchmarks focus on "needle-in-a-haystack" retrieval and ignore the temporal dynamics intrinsic to clinical reasoning.

Method

The benchmark is built on MIMIC-IV, filtering for patients with \(\geq 15\) hospitalizations (355 patients, 6,999 visits). A three-stage pipeline converts static EHRs into: 1. Event streams: each visit \(V_i = \{E_i^{adm}, E_i^1, \ldots, E_i^{dis}\}\) where each event \(E_i^k = \{a_i^k, t_i^k, p_i^k, o_i^k\}\) encodes action, timestamp, arguments, and result. 2. Three memory modules of increasing granularity: Note Memory (visit-level summaries \(M_N^{(i,j)}\)), Event Memory (chronological event sequence \(M_E^{(i,j)}\)), and Contextual Memory (LLM-style action–feedback dialog \(M_C^{(j,T)}\)). 3. Three evaluation task families: Factual QA (explicit timestamp vs. relative position retrieval), Temporal Reasoning (Visit Cloze / Visit Sorting / Joint Sorting, scored by Kendall's \(\tau\)), and Long-Horizon Decision Making with sub-tasks T3-N (Next Action Prediction), T3-A (Argument Prediction), and T3-D (Discharge Decision), using a time-decay scoring mechanism for actions within 24h of the ground-truth timestamp.

Evaluated models: gpt-5-mini, deepseek-v3.2 (+ thinking variant), qwen-turbo. Memory architectures: Naive Long-Context, RAG (text-embedding-3-small, Top-K retrieval), and Agent Memory (Mem0).

Key Contributions

  • LongMedBench dataset: 335 patients, avg. 19.72 visits/patient, 44.91 events/visit, derived from MIMIC-IV via a reproducible pipeline.
  • Tri-level memory architecture: Note, Event, and Contextual Memory modules enabling controlled evaluation at different granularities of historical access.
  • Progressive evaluation taxonomy: three task suites escalating from factual retrieval → temporal ordering → long-horizon planning, each with defined scoring metrics.
  • Empirical finding: decision-making accuracy is dominated by immediate context quality, not volume of retrieved history; memory augmentation (RAG, Mem0) yields minimal gains on decision tasks.

Results

Factual QA (qwen-turbo, explicit retrieval, Lab-T recall): - Window \(m=1\): 0.96; \(m=2\): 0.92; \(m=3\): 0.86; full history (\(m=\infty\)): 0.63 — severe degradation with scale. - Mem0 explicit Lab-T: 0.993 (near-perfect) but Medication imaging retrieval collapses (Img. 0.07 explicit); overall Mem0 average: 0.331.

Temporal Reasoning (Kendall's \(\tau\)): - Visit Cloze: gpt-5-mini 0.925, deepseek-v3.2-thinking 0.969, qwen-turbo 0.047. - Visit Sorting (no timestamps): best = deepseek-v3.2-thinking 0.424, gpt-5-mini 0.376. - Joint Sorting: best = deepseek-v3.2-thinking 0.330 — compounding difficulty confirmed.

Long-Horizon Decision Making (avg. across T3-A/D/N): - Event memory consistently outperforms note memory (e.g., gpt-5-mini event avg 0.478 vs. note avg 0.494 — marginal). - Ablation: adding 2–5 historical visits worsens performance vs. baseline (no memory): avg drops from 0.45 to 0.43–0.44. - RAG and Mem0 over event memory both yield avg ≈ 0.40–0.44, indistinguishable from the no-memory baseline (0.45).

Limitations

  • Benchmark is restricted to patients with \(\geq 15\) hospitalizations, which may skew toward complex chronic cases and limit generalizability.
  • All patients sourced from a single database (MIMIC-IV), limiting demographic and institutional diversity.
  • Decision-making ground truth is derived from what clinicians actually did, not necessarily optimal clinical actions.
  • The time-decay scoring for T3-N/T3-A is a proxy for clinical validity, not a verified clinician-rated assessment.
  • Context windows are capped at 128K tokens for most models, potentially artifactually truncating full-history conditions; the 400K gpt-5-mini condition is not fully ablated.
  • The paper notes that current tasks may need redesign to better expose deeper reasoning failures — the benchmark's sensitivity ceiling for decision tasks appears low.

Relevance to Agentic AI / LLM Agents

LongMedBench directly challenges the assumption that retrieval-augmented memory (RAG, Mem0) is sufficient for agentic reasoning over long horizons: the benchmark shows that decision-making performance saturates at the quality of immediate context regardless of how much historical information is made available, exposing a fundamental gap between memory retrieval and cross-session reasoning. This is directly relevant to agent memory architecture research — the result implies that architectural improvements to how agents integrate long-term history (not merely what is retrieved) are required. The temporal reasoning findings (strong on explicit timestamps, weak on implicit clinical progression) have implications for agent planning in any domain requiring causal and temporal inference over extended event sequences. The benchmark design pattern — converting real longitudinal records into multi-session event streams with hierarchical memory modules — is a reusable methodology for evaluating agentic systems in other long-horizon domains.