Skip to content

MEDDxAgent: A Unified Modular Agent Framework for Explainable Automatic Differential Diagnosis

🕒 Published (v1): 2025-02-26 14:31 UTC · Source: Arxiv · Venue: ACL 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MEDDxAgent is a modular multi-agent framework for interactive differential diagnosis (DDx) that uses a central orchestrator (DDxDriver) to coordinate a history-taking simulator and two specialized agents (knowledge retrieval, diagnosis strategy) in a ReAct-style iterative loop. It addresses the unrealistic assumption of complete upfront patient profiles by progressively gathering information across turns. Across GPT-4o, Llama-70B, and Llama-8B, it achieves >10 percentage-point GTPA@1 gains over single-turn baselines.

Problem

Existing LLM-based DDx systems are brittle in practice: they assume the full patient profile is available at inference time (single-turn paradigm), optimize only one diagnostic component in isolation, evaluate on a single dataset, and lack iterative refinement — all of which diverge from real clinical workflows where diagnoses evolve through multi-turn information gathering.

Method

MEDDxAgent wraps three components under a single orchestrator (DDxDriver):

  1. History Taking Simulator: Two LLMs role-play doctor and patient; the doctor (given only partial profile + DDxDriver-defined conversation goals) asks questions to elicit symptoms; the patient (given full profile) responds. Dialogue terminates on goal completion or a max-question cap (5/10/15 turns).

  2. Knowledge Retrieval Agent: DDxDriver issues a structured query; the agent extracts ≤3 medical keywords and searches Wikipedia and PubMed, synthesizing an evidence summary returned to the orchestrator.

  3. Diagnosis Strategy Agent: Produces a ranked DDx list in zero-shot, standard few-shot, or dynamic few-shot mode. Dynamic few-shot uses embedding-based retrieval (BioClinicalBERT or BGE) to select the most similar reference patients. Chain-of-Thought (CoT) can be overlaid on any setting, including CoT-annotated reference cases (à la MedPrompt).

DDxDriver (orchestrator) operates on the ReAct paradigm (thought → action → observation). It maintains the cumulative patient profile and ranked DDx, dispatches agent-specific instructions, logs all intermediate steps, and enforces stopping criteria. Iteration is either fixed (simulator → KR → DS in order) or dynamic (DDxDriver chooses the next module based on current evidence). Experiments run 1–3 outer iterations, each with 5 history-taking turns.

Evaluation metrics: GTPA@k (ground-truth pathology in top-k), average rank of correct diagnosis, and a new \(\Delta\text{Progress}\) metric:

\[\Delta\text{Progress} = \frac{1}{M}\sum_{i=1}^{M}\left(\frac{1}{N_i-1}\sum_{t=1}^{N_i-1}(r_{i,t} - r_{i,t+1})\right)\]

which measures per-iteration rank improvement toward the correct diagnosis.

Key Contributions

  • MEDDxAgent framework: modular orchestrator + history-taking simulator + knowledge retrieval + diagnosis strategy, with transparent interaction logging.
  • DDxDriver orchestrator: unified ReAct-based interface enabling iterative, explainable agent coordination with fixed and dynamic scheduling modes.
  • Multi-disease DDx benchmark: integrates DDxPlus (respiratory, 49 pathologies), iCraft-MD (skin, 394 diseases, interactive), and RareBench (421 rare diseases; RAMEDIS, MME, PUMCH subsets).
  • \(\Delta\text{Progress}\) metric: novel measure of iterative rank convergence for interactive DDx evaluation.
  • Empirical result: >10 pp GTPA@1 improvement over single-turn baselines for both large (70B) and small (8B) LLMs.

Results

  • GPT-4o, DDxPlus: MEDDx iter=3 reaches GTPA@1 = 0.86 vs. best single-turn baseline DS (n=5) = 0.72 (+14 pp); avg rank 1.29 vs. 2.14.
  • GPT-4o, iCraft-MD: MEDDx iter=2 GTPA@1 = 0.54 vs. DS (n=5) = 0.40 (+14 pp).
  • GPT-4o, RareBench: MEDDx iter=2 GTPA@1 = 0.56 vs. DS (n=5) = 0.50 (+6 pp).
  • Llama-70B, DDxPlus: MEDDx iter=2 GTPA@1 = 0.71 vs. DS (n=5) = 0.50 (+21 pp); \(\Delta\text{Progress}\) = +0.41 at iter=2.
  • Llama-8B, DDxPlus: MEDDx iter=3 GTPA@1 = 0.58 vs. DS (n=5) = 0.23 (+35 pp); \(\Delta\text{Progress}\) = +1.73 at iter=2.
  • PubMed retrieval slightly outperforms Wikipedia, especially on RareBench.
  • Dynamic few-shot with BAII embeddings best for DDxPlus and RareBench; zero-shot CoT best for iCraft-MD (distinct vignettes make few-shot noisy).
  • n=0 (no history taking) causes severe degradation: GPT-4o KR on RareBench drops from GTPA@1 = 0.45 (full profile) to 0.07.

Limitations

  • Experiments sample only 100 patients per dataset due to cost; results may not fully generalize.
  • Interactive iCraft-MD gains plateau after iter=1 for GPT-4o (GTPA@1 stagnates at 0.52–0.54), suggesting diminishing returns in skin disease DDx.
  • Llama-8B gains on iCraft-MD and RareBench remain modest (GTPA@1 ≤ 0.14), indicating small models struggle with complex disease spaces even with the full harness.
  • The history-taking simulator is LLM-based (not real patients), introducing simulation-to-reality gaps.
  • Only fixed and dynamic iteration strategies are explored; more sophisticated scheduling (e.g., learned policies) are left to future work.
  • PubMed retrieval restricted to commercially licensed full-text articles, potentially excluding relevant literature.

Relevance to Harnesses / Meta-Harnesses

MEDDxAgent is a direct instance of a domain-specific meta-harness: DDxDriver is a ReAct-based orchestrator that dynamically selects, sequences, and parametrizes specialized subagents (simulator, retriever, diagnoser) rather than treating any one LLM call as the solution. The fixed vs. dynamic iteration modes mirror the harness design choice between predetermined pipelines and adaptive agent scheduling. The framework's emphasis on centralized state management (patient profile), transparent logging of all intermediate steps, and configurable stopping criteria maps cleanly onto general meta-harness design patterns — making it a concrete medical case study for harness architects evaluating ReAct orchestration, modular agent composition, and iterative refinement strategies.