MedAgent-Pro: Towards Evidence-based Multi-modal Medical Diagnosis via Reasoning Agentic Workflow¶
🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link
Why this paper was selected
Evidence-based multi-modal medical diagnosis via structured agentic workflow [ICLR 2026]
Ask a follow-up
Open an assistant pre-loaded with this paper's context.
💬 Ask ChatGPT✦ Ask Claude
TL;DR¶
MedAgent-Pro is a hierarchical agentic workflow for multi-modal clinical diagnosis that replaces one-hop VQA with a two-level reasoning loop: disease-level RAG-guided plan generation followed by patient-level tool-orchestrated, evidence-reflective step-by-step reasoning. It achieves state-of-the-art performance across 50+ diseases and 10+ imaging modalities, outperforming GPT-4o by 34% bAcc on glaucoma and 21% on heart disease. The system is the first to enforce evidence-based reflection and risk-weighted decision-making inside a clinical agentic loop.
Problem¶
Existing VLMs and medical agentic systems treat diagnosis as empirical one-hop QA, producing direct answers without quantitative clinical evidence. Current medical agentic systems (e.g., MMedAgent, MDAgent) integrate tools in a fixed, disease-agnostic pipeline rather than dynamically orchestrating them according to the disease at hand, making them unable to compute required clinical indicators (cup-to-disc ratio, LVEF) or align with medical guidelines. This violates evidence-based medicine requirements for traceability and patient safety.
Method¶
MedAgent-Pro operates in two hierarchical stages:
Disease-level planning: A RAG agent \(R\) retrieves from a knowledge base \(K\) of 1,000+ MedlinePlus diseases (4,000+ NIH-authenticated guidelines). Articles are chunked at 300 tokens, embedded with PubMedBERT, and the top-5 chunks are retrieved via two-step retrieval (metadata summary filter → vector similarity). The VLM summarizes retrieved chunks into a procedural guideline \(G = V(R(K))\), then generates a JSON diagnostic plan \(P = \{P_1, \ldots, P_n\}\), where each step \(P_i\) specifies an action \(a_i \in A\), a toolset member \(t_i \in T\), and expected input/output types.
Patient-level reasoning: Memory is initialized by filtering \(P\) to only steps whose required inputs \(o_i\) exist in the patient's data \(D\): \(M = \{P_i \in P \mid o_i \in D\}\). At each step, the VLM invokes the appropriate tool (MedSAM segmentation, Cellpose, or an LLM-based coding module) for quantitative analysis, or itself for qualitative analysis. An evidence-based reflection mechanism assigns each output \(r_i\) a status:
where \(\phi\) is a VLM-implemented reliability assessor. Reliable intermediate results become evidence feeding the next step; unreliable ones halt that branch. Final risk-weighted diagnosis is computed as \(\rho = \sum_i w_i r_i\) against a threshold \(\theta\), with weights \(W\) assigned by the VLM from clinical importance in \(G\).
Key Contributions¶
- First agentic paradigm enforcing evidence-based, step-by-step clinical reasoning with full guideline traceability, replacing empirical one-hop QA.
- Hierarchical workflow separating disease-level RAG-guided standardized planning from patient-level personalized tool-orchestrated reasoning.
- Evidence-based reflection mechanism with three-state step evaluation (Continue/Terminate/Complete) and adaptive memory adjustment.
- Risk-based decision module using VLM-assigned clinical weights over verified indicator sets.
- Evaluation on 10+ imaging modalities, 20+ anatomical regions, 50+ diseases (REFUGE2, MITEA, MIMIC, NEJM).
Results¶
- Glaucoma (REFUGE2): 90.4% bAcc, 76.4% F1 vs. GPT-4o (56.4% bAcc, 21.1% F1); +34.0% bAcc, +55.3% F1. Surpasses REFUGE2 challenge winner VUNO EYE TEAM (88.3 AUC) with 95.1 AUC.
- Heart disease (MITEA): 77.8% bAcc, 72.3% F1 vs. GPT-4o (56.8% bAcc, 28.1% F1); +21.0% bAcc, +44.2% F1.
- Chest X-ray (MIMIC, 12 tasks): 72.0% avg. bAcc vs. GPT-4o 58.3%; surpasses specialist CheXagent (69.1%) and Maira-2 (64.1%).
- NEJM (992 real-world cases): +7.9% overall accuracy over GPT-4o; larger gains on tool-supported domains (cell imaging, ophthalmology, CT/MRI).
- Outperforms all tested medical agentic systems (MedAgents, MMedAgent, MDAgent) across all benchmarks.
Limitations¶
- The toolset is domain-specific; non-clinical images (e.g., daily photographs in NEJM) lack compatible tool support, reducing gains in those cases.
- Disease-level plans are pre-generated per disease and stored as JSON; adapting to rare or novel diseases requires new guideline coverage in the knowledge base.
- Relies on GPT-4o as the backbone VLM; performance dependency on a proprietary, closed model complicates reproducibility and cost.
- 3D modality handling (echocardiography) uses slice sampling with averaging, which is an approximation rather than native volumetric processing.
- Threshold \(\theta\) for risk-based decision and reliability function \(\phi\) are not explicitly ablated in detail in the main text.
Relevance to Harnesses / Meta-Harnesses¶
MedAgent-Pro is a domain-specific meta-harness that composes heterogeneous specialized tools (segmentation models, coding modules, RAG, VLM) under a runtime-generated, guideline-aligned plan, directly instantiating the harness pattern of dynamic tool orchestration around a central LLM orchestrator. The disease-level planner acts as a meta-layer that configures the patient-level execution harness per task instance, demonstrating how harness logic can be externalized into a retrieved knowledge graph rather than hard-coded. The evidence-based reflection loop is a concrete implementation of intra-harness self-verification: the harness evaluates each sub-step's output before deciding whether to propagate, terminate, or complete, which generalizes to quality-gated pipelines in non-medical harnesses. For researchers tracking harness design, this paper offers a clinical existence proof that RAG-configured, reflection-gated, multi-tool harnesses substantially outperform both monolithic VLMs and fixed-pipeline agentic systems.