Skip to content

MAM: Modular Multi-Agent Framework for Multi-Modal Medical Diagnosis via Role-Specialized Collaboration

🕒 Published (v1): 2025-06-24 17:52 UTC · Source: Arxiv · Venue: ACL 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MAM is a five-role modular multi-agent framework for multimodal medical diagnosis, decomposing the diagnostic process across a General Practitioner, Specialist Team, Radiologist, Medical Assistant, and Director agent. Rather than retraining a single unified model, each agent can be updated independently, and agents collaborate via iterative voting-based consensus to produce a final diagnosis. Evaluated on 10 public datasets spanning text, image, audio, and video, MAM achieves 18%–365% improvement over modality-specific baselines.

Problem

Unified multimodal medical LLMs require expensive global retraining for any knowledge update, lack modularity, and must exhibit broad cross-modal competence in a single model. This limits adaptability and makes integrating specialist domain knowledge costly.

Method

MAM orchestrates five LLM-based agents in a fixed pipeline:

  1. General Practitioner classifies disease type \(d = C^G(M)\) from multimodal input \(M\) and routes the case to the relevant Specialist Team \(S = R^G(d)\).
  2. Specialist Team decomposes the problem into subproblems \(P = D^S(M)\) and anonymizes them.
  3. Medical Assistant retrieves external knowledge via Google API on the anonymized subproblems and summarizes it: \(I_s = \text{Summarize}_A(\text{Retrieve}_A(P))\).
  4. Specialist Team + Radiologist independently generate diagnostic opinions \(O_{s_i} = \text{Diag}^{s_i}(M, I_s)\) and \(O_\text{Rad} = \text{Diag}^\text{Rad}(M)\).
  5. Director synthesizes a report \(R_p = \text{Synth}^\text{Dir}(\{O_{s_i}\}, O_\text{Rad}, I_s)\); specialists vote \(v_i \in \{0,1\}\); if \(V = \sum v_i = n\) (unanimous), the Director derives \(D_\text{final} = \text{Diagnosis}^\text{Dir}(R_p)\). Otherwise, the discussion loop repeats up to 3 rounds.

The empirical motivation is established by two preliminary studies showing: (a) role-assigned prompting improves accuracy by 5–38.5 pp over vanilla prompting, and (b) LLMs can reason above chance to identify the correct diagnosis from a generated candidate set.

Key Contributions

  • A modular five-role multi-agent pipeline (GP, Specialist Team, Radiologist, Medical Assistant, Director) enabling per-agent knowledge updates without global retraining.
  • Voting-based iterative consensus loop (max 3 rounds) for synthesizing multi-specialist opinions.
  • External knowledge retrieval via anonymized sub-problem queries integrated as a Medical Assistant role.
  • Empirical motivation study demonstrating the independent value of role assignment and diagnostic discernment in LLMs.
  • Ablation confirming additive contribution of: role specialization → inter-agent discussion → retrieval augmentation.

Results

  • Text: MAM achieves 40.0 (MedQA) and 84.0 (PubMedQA) vs. best baseline Medichat-Llama3-8B at 48.5/30.8 — gains of ~74% and ~173%.
  • Image: MAM reaches 47.6 (PathVQA), 32.5 (PMC-VQA), 35.1 (DeepLesion), 58.6 (NIH), 97.9 (Brain Tumor) vs. HuatuoGPT-Vision-7B at 40.1/24.0/11.1/12.6/80.2.
  • Audio: MAM achieves 64.0 (Heartbeat) and 47.9 (SoundDr) vs. Qwen-Audio-Chat at 34.9/25.0.
  • Video: MAM achieves 74.3 (MedVidQA) vs. VideoLLaMA2-7B at 55.3.
  • Overall claimed improvement range: 18%–365% over baseline models.
  • Ablation (Table 7): each component adds monotonically; retrieval provides the largest single jump on several datasets (e.g., NIH: 38.6→58.6).

Limitations

  • External knowledge retrieval uses Google API as a proxy for a real hospital database; clinical deployment would require integration with actual medical knowledge bases.
  • Unanimity-based consensus (\(V = n\)) with \(n = 3\) specialists and max 3 discussion rounds may be too strict or too loose for real clinical scenarios; the paper does not analyze failure modes when consensus is never reached.
  • All agents are existing off-the-shelf LLMs (7B–13B range); no new model training is performed, so performance is bounded by the underlying models' capabilities.
  • Evaluation uses accuracy on multiple-choice/yes-no framed datasets, which may not reflect free-form clinical diagnostic quality.
  • Dataset sizes are small (200–461 cases per dataset), limiting statistical robustness of the reported gains.

Relevance to Harnesses / Meta-Harnesses

MAM is a concrete medical-domain meta-harness: it wraps heterogeneous specialist LLMs in a coordinator that handles routing, retrieval, iterative deliberation, and consensus arbitration — precisely the orchestration patterns central to harness design. The Director agent's role as a synthesis-and-halt controller mirrors meta-harness "judge" or "aggregator" nodes, and the voting-based termination condition is a domain-specific instance of the convergence criteria harnesses must implement. The ablation study cleanly isolates the contribution of each harness layer (role assignment, discussion loop, retrieval), providing empirical grounding for harness component design. For researchers building general meta-harnesses, MAM demonstrates how domain-specific role decomposition, external knowledge injection, and iterative voting can be composed into a reproducible orchestration pattern.