Skip to content

MMedAgent-RL: Optimizing Multi-Agent Collaboration for Multimodal Medical Reasoning

🕒 Published (v1): 2025-05-31 13:22 UTC · Source: Arxiv · Venue: ICLR 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MMedAgent-RL is an RL-trained multi-agent framework for multimodal medical VQA that replaces static GP→Specialist→GP pipelines with dynamically optimized agents. Two Qwen2.5-VL-based GP agents (triage doctor and attending physician) are trained via GRPO, with the attending physician using a three-stage curriculum learning strategy that modulates policy entropy based on specialist reliability. It achieves an average 23.6% improvement over SFT baselines across five medical benchmarks.

Problem

Existing medical multi-agent systems (MedAgents, MDAgents) implement fixed, predetermined GP→Specialist→GP workflows. These static pipelines cannot adapt reasoning dynamically: the triage doctor uses rigid assignment rules, and the attending physician cannot learn when to trust vs. override specialist opinions, especially when specialists are partially or wholly wrong—introducing noise that SFT cannot handle.

Method

Stage 1 — Triage Optimization: The triage doctor (\(a^{\text{triage}}_{\text{GP}}\)) is fine-tuned with GRPO using image-modality ground-truth labels (e.g., pathology slide → pathologist) to select among \(k=7\) specialties. Reward: \(R_{\text{format}} \in \{0, 0.5\}\) plus \(R_{\text{accuracy}} \in \{0, 1\}\).

Stage 2 — Specialist Consultation: \(e=3\) proprietary LVLMs (GPT-4o family) role-play specialists in the triaged department and independently produce opinions \(y_d\). No inter-specialist interaction occurs.

Stage 3 — Curriculum-Based Entropy-Aware RL (C-MARL): Training data is partitioned by specialist accuracy \(s = \text{Acc}(y_d, y^*)\) into \(D_{\text{easy}}\) (\(s=1\)), \(D_{\text{medium}}\) (\(0<s<1\)), \(D_{\text{hard}}\) (\(s=0\)). The attending physician is trained sequentially on these splits using GRPO augmented with a dynamic entropy bonus:

\[J_{\text{C-MARL}}(\theta) = \mathbb{E}\left[J_{\text{GRPO}}(\theta) + \gamma_s \cdot H_t(\pi_\theta^{\text{attend}})\right]\]

where \(\gamma_{\text{easy}} \approx 0\), \(\gamma_{\text{medium}} > 0\), \(\gamma_{\text{hard}} \gg \gamma_{\text{medium}}\). Low entropy is rewarded when specialists are reliable (exploit); high entropy is incentivized when specialists are wrong (explore/correct).

Key Contributions

  • C-MARL: Curriculum-based entropy-aware RL for multi-agent collaboration, with difficulty stratified by specialist reliability rather than problem formulation.
  • Stage-wise RL training of two GP agents in a clinical-simulation pipeline, replacing static rule-based coordination.
  • Theoretical justification: Formal convergence analysis (Theorem 4.1) showing curriculum learning requires fewer iterations than standard SGD under the same error tolerance, with the gap scaling as \(\sum_{j=0}^{J-1} \log \|\theta_j^* - \theta_{j+1}^*\|_2^2\).
  • Empirical generalization: Strong performance on both in-domain and out-of-distribution medical VQA datasets with a 7B-parameter backbone.

Results

  • In-domain average (VQA-RAD, SLAKE, PathVQA): MMedAgent-RL (7B) = 73.3% vs. best multi-agent baseline AFlow = 67.5% (+11%); vs. GPT-4o single-agent = 68.6% (+7%).
  • Out-of-distribution average (OmniMedVQA + MMMU-Med): MMedAgent-RL = 72.6% vs. AFlow = 56.6% (+28%); vs. GPT-4o = 69.1% (+5%).
  • With test-time scaling (majority voting): in-domain 76.1% (+14% over AFlow), OOD 76.6% (+18%).
  • Over SFT baselines: +23.6% average performance gain.
  • MMedAgent-RL outperforms all listed open-source models (up to 34B params) and proprietary GPT-4o on in-domain benchmarks.

Limitations

  • Specialist doctors are proprietary LVLMs (GPT-4o family), introducing latency and cost not present in fully open pipelines.
  • Only 7 predefined specialties; coverage of rare or interdisciplinary domains is untested.
  • Curriculum difficulty labels are derived from specialist accuracy on training data—if specialists are systematically biased on a new domain, the curriculum stratification may be miscalibrated.
  • Out-of-distribution evaluation uses only two datasets; broader generalization to non-VQA clinical tasks (radiology report generation, clinical notes) is not assessed.
  • Theoretical convergence guarantee relies on regularity assumptions (Assumptions B.2–B.6) not verified empirically.

Relevance to Harnesses / Meta-Harnesses

MMedAgent-RL is a concrete instantiation of a trainable meta-harness: rather than hard-coding the orchestration logic (which agent to call, how to weight outputs), it learns the coordination policy end-to-end via RL. The curriculum-stratified training loop—where the harness itself determines task difficulty and schedules agent interactions—is precisely the kind of adaptive orchestration layer that meta-harness research aims to generalize. The C-MARL algorithm addresses a core meta-harness problem: how a controller agent should dynamically trust, override, or seek consensus from sub-agents whose reliability varies. This work provides both an empirical template and a theoretical convergence argument for RL-trained harnesses operating over heterogeneous, unreliable sub-agents.