Skip to content

MMedAgent-RL: Optimizing Multi-Agent Collaboration for Multimodal Medical Reasoning

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

RL-optimized multi-agent collaboration for multimodal medical reasoning [ICLR 2026]

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MMedAgent-RL is a reinforcement learning-driven multi-agent framework for multimodal medical reasoning that replaces static GP→Specialist→GP pipelines with RL-trained agents capable of dynamic, adaptive collaboration. A triage doctor and attending physician are trained via GRPO on Qwen2.5-VL; the attending physician is further optimized with Curriculum-Based Multi-Agent Reinforcement Learning (C-MARL), which uses dynamic entropy regulation to handle unreliable specialist outputs. The system achieves 23.6% average accuracy improvement over supervised fine-tuning baselines across five medical VQA benchmarks.

Problem

Prior multi-agent medical AI systems (MedAgents, MDAgents) simulate clinical workflows as fixed, predetermined pipelines—GP routes to specialists, specialists respond, GP aggregates—but these static harnesses cannot adapt when specialist outputs are noisy, conflicting, or incorrect. Single-agent Med-LVLMs also fail to generalize across specialties. No prior work trains the orchestrating agents themselves via RL to learn when to trust vs. override specialist consensus.

Method

Two GP agents are built on Qwen2.5-VL and trained sequentially with GRPO:

  1. Triage doctor: Given \(k=7\) candidate specialties, GRPO optimizes department selection using image-modality ground-truth labels (e.g., pathology slide → pathologist). Reward: \(R_{\text{format}} \in \{0, 0.5\}\) + \(R_{\text{accuracy}} \in \{0, 1\}\).

  2. Specialists: Proprietary LVLMs (GPT-4o) role-play \(e=3\) specialists for the assigned department; they are not trained but generate reference opinions \(y_d\).

  3. Attending physician (C-MARL): The training corpus is partitioned by specialist accuracy \(s = \text{Acc}(y_d, y^*)\) into three difficulty levels—easy (\(s=1\)), medium (\(0 < s < 1\)), hard (\(s=0\))—and the model is trained on them in curriculum order. The GRPO objective is augmented with a dynamic entropy bonus: $\(J_{\text{C-MARL}}(\theta) = \mathbb{E}\!\left[J_{\text{GRPO}}(\theta) + \gamma_s \cdot H_t(\pi_\theta^{\text{attend}})\right]\)$ where \(H_t = -\sum_{j=1}^V p_{t,j}\log p_{t,j}\) and the coefficient \(\gamma_s\) scales from \(\gamma_{\text{easy}} \approx 0\) (exploit reliable specialists) to \(\gamma_{\text{hard}} \gg \gamma_{\text{medium}}\) (explore away from wrong specialists). This controls the exploration-exploitation trade-off adaptively per training sample.

Key Contributions

  • C-MARL: Curriculum-based RL with dynamic entropy regulation for training an attending-physician agent to selectively trust or override specialist opinions of varying reliability.
  • Stage-wise RL training of two orchestrating GP agents (triage and attending) within a multi-agent harness, leaving specialist models frozen.
  • Theoretical convergence analysis (Theorem 4.1) showing curriculum learning reaches the optimal policy with fewer total iterations than standard SGD when consecutive stage optima are close.
  • Test-time scaling via majority voting over diverse rollouts, further boosting performance by 4.5%.

Results

  • In-domain average (VQA-RAD, SLAKE, PathVQA): MMedAgent-RL (7B) 73.3% vs. best multi-agent baseline AFlow 67.5% (+8.6 pp); vs. GPT-4o single-agent 68.6% (+4.7 pp).
  • Out-of-distribution average (OmniMedVQA, MMMU-Med): 72.6% vs. AFlow 56.6% (+16.0 pp); vs. GPT-4o 69.1% (+3.5 pp).
  • vs. SFT baseline: +23.6% average gain cited in abstract.
  • With test-time scaling: in-domain avg rises to 76.1% (+14% over AFlow); OOD avg to 76.6% (+18% over AFlow).
  • Outperforms MedAgents, MDAgents, AFlow across all six datasets.

Limitations

  • Specialists are proprietary LVLMs (GPT-4o), introducing inference cost and external API dependency; specialist quality directly bounds the attending physician's ceiling.
  • Only multiple-choice VQA evaluated; generalizability to open-ended generation or report writing is unverified.
  • Curriculum difficulty labels are derived from specialist accuracy on training data, creating a dependency on specialist quality for data curation as well as inference.
  • The 7-specialty taxonomy is fixed; expanding to more granular or overlapping departments requires redesign.
  • Theoretical convergence guarantees assume regularity conditions (Assumptions B.2–B.6) that may not hold in practice.

Relevance to Harnesses / Meta-Harnesses

MMedAgent-RL is a concrete example of a trainable meta-harness: rather than hard-coding agent routing and aggregation logic, it learns optimal orchestration (triage assignment, specialist integration weights, when to deviate from consensus) directly from reward signals. The C-MARL curriculum is directly analogous to meta-harness scheduling—deciding in what order and with what exploration budget sub-agents should be invoked based on estimated subtask difficulty. For harness researchers, this demonstrates that replacing static routing rules with RL-trained controller agents meaningfully outperforms hand-designed multi-agent pipelines, and the dynamic entropy mechanism offers a principled way to control sub-agent exploration within a composed system.