Can LLM Agents Generate Real-World Evidence? Evaluating Observational Studies in Medical Databases

🕒 Published (v1): 2026-03-24 03:50 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

RWE-bench is a benchmark of 162 peer-reviewed observational studies grounded in MIMIC-IV that requires LLM agents to execute full end-to-end clinical analyses—cohort construction, statistical modeling, and reporting—and submit tree-structured evidence bundles evaluated at both question and task levels. The best agent configuration (RWEAgent + Claude-Sonnet-4) achieves only 39.9% task success rate, and agent scaffold choice alone causes over 30% variation in performance.

Problem

Existing LLM agent evaluations of medical data analysis either assume preprocessed inputs, test only isolated subtasks, or judge only final answers—missing the integrity and causal structure of a complete evidence bundle. No prior benchmark evaluates whether agents can execute a full observational study on a raw, institutionally realistic clinical database (unreduced schema, no task-specific preprocessing) while reproducing the methodological decisions of peer-reviewed work.

Method

Each benchmark task provides: a study protocol \(P\) (background, methods), the MIMIC-IV database \(D\), and a two-stage question set \(Q = Q^{(0)} \cup Q^{(1)}\). \(Q^{(0)}\) consists of multiple-choice verification questions; \(Q^{(1)}\) fill-in-the-blank quantitative fields are conditionally unlocked only upon correct \(Q^{(0)}\) responses (gating). Agents write and execute R code against a PostgreSQL MIMIC-IV instance (Docker-sandboxed), submit a cohort table, and iteratively produce an evidence bundle \(\hat{A}\).

Three agent scaffolds are evaluated: MLAB (ReAct-style with structured reflection stages), OpenHands (CodeAct, general-purpose coding agent), and RWEAgent (adapted from EHRAgent for long-horizon clinical execution). Six LLMs are tested: GPT-4.1, O4-mini, Claude-Sonnet-4 (closed-source); Qwen3-30B-A3B, MiniMax-M2.1, GLM-4.7 (open-source).

Evaluation metrics: - ACC: multiple-choice accuracy on \(Q^{(0)}\) - RAR (Regulatory Alignment Rate): binary metric checking that agent and publication agree on effect direction and CI overlap with null: \(\text{RA} = \mathbb{I}\!\left[(\hat{r}-1)(r-1)>0 \;\wedge\; (1 \in [\hat{l},\hat{u}]) \Leftrightarrow (1 \in [l,u])\right]\) - SMR (Significance Match Rate): whether agent and publication agree on statistical significance at \(\alpha=0.05\): \(\text{SM} = \mathbb{I}[(\hat{p}<0.05) \Leftrightarrow (p<0.05)]\) - SR (Success Rate): all choice questions correct AND all statistical fields semantically aligned - CR (Completion Rate), Steps: execution efficiency metrics

An automated cohort evaluation pipeline uses LLM judges backed by a human-annotated set to rapidly localize errors in constructed cohorts.

Key Contributions

• RWE-bench: 162 end-to-end observational study tasks derived from PubMed papers on MIMIC-IV, with hierarchical (tree-structured) evidence bundle evaluation
• Multi-scaffold, multi-model evaluation: 6 LLMs × 3 agent scaffolds, systematically quantifying the independent effect of scaffold choice
• Gated hierarchical evaluation: two-stage question structure that exposes intermediate failure modes rather than only final-answer accuracy
• Automated cohort verification: rule-integrated LLM-judge pipeline for rapid cohort error localization without per-task human review
• RWE-bench-hard: harder subset (≥9 fields/task; best SR 23%) for cost-efficient evaluation with stronger stratification

Results

• Best overall: RWEAgent + Claude-Sonnet-4, SR = 39.9%
• Best open-source: RWEAgent + MiniMax-M2.1, SR = 30.4%
• Best question-level ACC: RWEAgent + Claude-Sonnet-4 at 71.2%; best RAR: 60.3%; best SMR: 55.7%
• Agent scaffold effect: GPT-4.1 achieves SR ~35% under RWEAgent, ~10% under MLAB, ~1% under OpenHands—over 30% absolute variation from scaffold alone
• OpenHands + MiniMax-M2.1 consumes >1.4M tokens/task with no corresponding performance gain
• Medical fine-tuned models (e.g., MedCopilot-14B) failed to sustain instruction-following under long-horizon coding; effective performance appears to require large parameter scale
• RWE-bench-hard (tasks with ≥9 fields) reduces best SR to 23%, providing stronger model stratification at lower evaluation cost

Limitations

• MIMIC-IV only; generalizability to other EHR systems (Epic, Optum, claims data) is untested
• Single database version (v2.2); schema changes could invalidate existing tasks
• Task success requires exact reproduction of published results, which may penalize valid alternative analytical choices
• All scaffolds are general-purpose; no scaffold was purpose-built for observational research, so the ceiling may not reflect what specialized designs could achieve
• Temperature fixed at 0.6 introduces stochastic variance; each task run only 3 times
• Evaluation of descriptive statistics fields relies on multiple-choice proxies rather than direct numerical alignment

Relevance to Harnesses / Meta-Harnesses

RWE-bench is directly a benchmark harness for LLM agent scaffolds: it standardizes a plug-and-play Docker environment (database + R workspace containers), defines parameterized task inputs \((P, D, Q)\), enforces resource constraints (100 steps, 20 min/step, 2 h total), and provides layered automated evaluation pipelines—making it a concrete example of a domain-specific meta-harness for agent assessment. The finding that scaffold choice alone causes >30% performance variation is a key empirical result for harness designers: it implies that the execution framework (ReAct vs. CodeAct vs. task-adapted) is at least as important as the base model, and that meta-harnesses must control scaffold configuration as a first-class experimental variable. The automated cohort evaluation pipeline—a rule-integrated LLM-judge layer sitting above the agent execution loop—is itself an instance of a verification sub-harness, relevant to work on compositional harnesses that wrap agents with automated correctness checks.