MedAgent-Pro: Towards Evidence-based Multi-modal Medical Diagnosis via Reasoning Agentic Workflow

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Evidence-based multi-modal medical diagnosis via staged agentic reasoning workflow

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MedAgent-Pro is a hierarchical agentic workflow for multi-modal medical diagnosis that mirrors clinical practice through evidence-based, guideline-aligned reasoning rather than one-hop VQA. It combines a RAG agent for disease-level planning with tool-augmented, reflection-driven patient-level reasoning to produce quantitatively grounded diagnoses. It substantially outperforms general VLMs and prior medical agentic systems across 50+ diseases and 10+ imaging modalities.

Problem

Existing VLMs and medical agentic systems treat diagnosis as an empirical one-hop QA task — generating conclusions from internal knowledge without structured clinical reasoning, quantitative indicator computation, or evidence traceability. Current agentic systems integrate tools in a fixed pipeline rather than dynamically orchestrating them per disease, violating the principle of evidence-based medicine and limiting clinical usability.

Method

MedAgent-Pro employs a two-level hierarchical workflow:

Disease-level planning: A RAG agent \(R\) retrieves relevant content from a knowledge base \(\mathcal{K}\) (MedlinePlus; 1,000+ diseases, 4,000+ NIH/NLM-authenticated guidelines) via two-stage retrieval — metadata filtering followed by PubMedBERT-embedded chunk similarity search (top-5 chunks, 300-token chunks). A VLM \(V\) summarizes retrieved content into a procedural guideline \(G = V(R(\mathcal{K}))\) and generates a disease-specific JSON diagnostic plan \(P = \{P_1, \ldots, P_n\}\), where each step \(P_i\) specifies an action \(a_i \in \mathcal{A}\) drawn from a toolset \(\mathcal{T}\).

Patient-level reasoning: For each patient's multi-modal data \(\mathcal{D}\), available steps are selected into memory \(M = \{P_i \in P \mid o_i \in \mathcal{D}\}\). The VLM orchestrates tool-based actions (segmentation models MedSAM/Cellpose, LLM-based coding tools for numerical computation, VLM for qualitative analysis). After each step \(P_i'\), an evidence-based reflection assigns status \(s_i \in \{\text{Continue}, \text{Terminate}, \text{Complete}\}\) based on output reliability \(\phi(r_i, o_i, G)\):

\[s_i = \begin{cases} \text{Complete} & r_i \in I \\ \text{Terminate} & r_i \notin I \wedge \neg\phi(r_i, o_i, G) \\ \text{Continue} & r_i \notin I \wedge \phi(r_i, o_i, G) \end{cases}\]

Final diagnosis uses a weighted risk score \(\rho = \sum_{i} w_i r_i\) over completed indicator set \(R_{\text{final}}\), with weights \(W = V(R_{\text{final}} | G)\) assigned per clinical importance, compared against a threshold \(\theta\).

Key Contributions

• First agentic medical diagnosis paradigm producing systematic, evidence-backed reasoning aligned with modern clinical workflow rather than empirical VQA outputs.
• Hierarchical architecture separating guideline-driven disease-level planning (RAG-based) from personalized patient-level reasoning (tool-augmented, reflection-guided).
• Evidence-based reflection mechanism that evaluates step-by-step output reliability and gates propagation, preventing error cascades in multi-step reasoning chains.
• Broad empirical validation across 10+ imaging modalities, 20+ anatomies, 50+ diseases, including clinician expert evaluation.

Results

• Glaucoma (REFUGE2): MedAgent-Pro bAcc 90.4%, F1 76.4% vs. GPT-4o bAcc 56.4%, F1 21.1%; vs. MDAgent (best prior agentic) bAcc 56.8%, F1 22.2%. Also surpasses REFUGE2 challenge winners (AUC 95.1% vs. 88.3%).
• Heart disease (MITEA): MedAgent-Pro bAcc 77.8%, F1 72.3% vs. GPT-4o bAcc 51.1%, F1 15.9%; vs. MDAgent bAcc 57.2%, F1 30.3%.
• Chest X-ray (MIMIC, 12 thoracic conditions): MedAgent-Pro avg. bAcc 72.0% vs. GPT-4o 58.3%, CheXagent 69.1%.
• NEJM real-world cases (992 cases): MedAgent-Pro outperforms GPT-4o by 7.9% overall; larger gains in tool-supported modalities (cell imaging, CT/MRI, ophthalmology).
• Surpasses task-specific ophthalmology VLM VisionUnite (bAcc 90.4% vs. 85.8%) and chest X-ray specialist Maira-2 (bAcc 72.0% vs. 64.1%).

Limitations

• Relies on GPT-4o as the base VLM orchestrator; performance with open-source or smaller VLMs is not systematically characterized.
• Performance degrades in domains lacking compatible visual tool support (non-clinical images); the system cannot compensate for absent tool coverage.
• RAG knowledge base is limited to MedlinePlus; rarer diseases or conditions not covered may produce less reliable plans.
• The evidence-based reflection function \(\phi\) is implemented by the VLM itself, introducing potential circular reasoning where the same model assesses its own outputs.
• 3D echocardiography is handled via 2D slice sampling (3 random slices × 10 repetitions), which may lose volumetric information.

Relevance to Agentic AI / LLM Agents

MedAgent-Pro exemplifies a principled design pattern for domain-specific LLM agents: separating high-level plan generation (guided by external knowledge via RAG) from low-level plan execution (tool invocation with per-step reflection), rather than letting the LLM do both in one pass. The evidence-based reflection mechanism — which gates information flow between steps based on reliability rather than always propagating intermediate results — is a concrete contribution to multi-step agent robustness, generalizable beyond medicine. The work directly challenges the "agentic systems aggregate tools in a fixed manner" failure mode, demonstrating that dynamic, condition-dependent tool orchestration driven by retrieved domain knowledge produces substantially better outcomes in high-stakes settings.