Skip to content

MMedAgent-RL: Optimizing Multi-Agent Collaboration for Multimodal Medical Reasoning

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

RL-optimized multi-agent collaboration for multimodal medical reasoning across specialties

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MMedAgent-RL replaces the static GP→Specialist→GP multi-agent pipeline in medical VQA with a reinforcement-learning-trained framework where both triage and attending-physician agents are dynamically optimized via GRPO. A curriculum learning strategy with dynamic entropy regulation (C-MARL) teaches the attending physician when to trust versus override specialist consensus. On five medical VQA benchmarks the system achieves an average 23.6% gain over strong supervised fine-tuning baselines.

Problem

Existing multi-agent medical reasoning systems (MedAgents, MDAgents) simulate clinical workflows with a fixed GP→Specialist→GP sequence, but the inter-agent interaction pattern is static and predetermined: neither the triage doctor nor the attending physician is trained to dynamically adapt. Specialist outputs are often inconsistent or wrong, introducing noise that static pipelines cannot handle without either blindly trusting or blindly ignoring expert opinions.

Method

Stage 1 — Triage doctor optimization. A Qwen2.5-VL-based triage GP is fine-tuned with GRPO, using image-modality ground-truth labels (e.g., pathology slide → pathologist) as the reward signal. It selects among \(k=7\) specialties via \(d = \arg\max_k \pi_{\theta^{\text{triage}}}(k|x)\).

Stage 2 — Specialist consultation. Proprietary LVLMs (GPT-4o) role-play \(e=3\) specialists per sample and independently generate answers; no inter-specialist communication occurs to avoid majority-vote suppression of minority-correct opinions.

Stage 3 — Attending physician via C-MARL. The attending physician (second Qwen2.5-VL GP) is trained with Curriculum-Based Multi-Agent RL. Training data is stratified by specialist accuracy \(s = \text{Acc}(y_d, y^*)\): fully correct (\(s=1\)) → easy; partially correct (\(0<s<1\)) → medium; fully wrong (\(s=0\)) → hard. Each stage uses GRPO augmented with a dynamic entropy bonus:

\[J_{\text{C-MARL}}(\theta) = \mathbb{E}\!\left[J_{\text{GRPO}}(\theta) + \gamma_s \cdot H_t(\pi_\theta^{\text{attend}})\right]\]

where \(H_t = -\sum_{j=1}^V p_{t,j} \log p_{t,j}\) and \(\gamma_s\) escalates from \(\gamma_{\text{easy}} \approx 0\) (exploit reliable consensus) to \(\gamma_{\text{hard}} \gg \gamma_{\text{medium}}\) (aggressively explore when specialists are all wrong). Test-time scaling uses majority voting over multiple rollouts.

Key Contributions

  • MMedAgent-RL: first RL-trained multi-agent framework for multimodal medical reasoning, making both the triage and attending-physician roles trainable rather than frozen.
  • C-MARL: curriculum learning stratified by specialist reliability with dynamic entropy regulation, formally addressing the exploration–exploitation dilemma in multi-agent integration.
  • Theoretical guarantee: Theorem 4.1 shows curriculum learning converges with lower sample complexity than standard SGD, with the gap driven by the sum of log-distances between consecutive curriculum-stage optimal policies.
  • Empirical validation: 23.6% average accuracy gain over SFT baselines; 14–18% gain over best prior multi-agent systems on out-of-distribution datasets.

Results

  • In-domain (VQA-RAD, SLAKE, PathVQA) average: MMedAgent-RL 73.3% vs. best multi-agent baseline AFlow 67.5% (+11%); vs. GPT-4o 68.6% (+7%).
  • Out-of-distribution (OmniMedVQA + MMMU-Med) average: 72.6% vs. AFlow 56.6% (+28% relative); vs. GPT-4o 69.1% (+5%).
  • With test-time scaling (majority voting): in-domain 76.1% (+14% over AFlow); OOD 76.6% (+18% over AFlow).
  • Outperforms all open-source single-agent models (LLaVA-v1.6-34B: 61.6% in-domain) and matches or exceeds GPT-4o on all subsets.
  • Per-dataset: VQA-RAD 71.5%, SLAKE 76.2%, PathVQA 72.3% (each +10–12% over AFlow).

Limitations

  • Specialist doctors are proprietary LVLMs (GPT-4o), introducing API cost and latency; the system is not fully open-source.
  • Only multiple-choice VQA is evaluated; open-ended generation or report-generation tasks are untested.
  • Curriculum difficulty is defined solely by specialist accuracy, not by intrinsic question hardness; curriculum effectiveness may depend on how well proprietary specialists cover each specialty.
  • The 7-specialty taxonomy is coarse and may not generalize to highly granular sub-specialties not covered by training data.
  • Theoretical analysis assumes regularity conditions (Lipschitz gradients, bounded noise) that may not hold in practice with large language model policies.

Relevance to Agentic AI / LLM Agents

MMedAgent-RL is a direct contribution to the challenge of training multi-agent systems end-to-end rather than treating agent roles as fixed prompting strategies. The C-MARL framework addresses a fundamental problem in any orchestrator-worker architecture: how should an aggregator learn to weight, trust, or override worker outputs when those outputs vary in reliability? The curriculum-plus-entropy approach is domain-agnostic in principle and could transfer to any agentic pipeline where a controller must integrate noisy specialist advice. The paper also demonstrates that GRPO, originally developed for single-model reasoning, can be extended to multi-role workflows by training each role independently on role-appropriate reward signals.