Skip to content

Unlocking the Power of Multi-Agent LLM for Reasoning: From Lazy Agents to Deliberation

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR; multi-agent deliberation loop overcomes lazy-agent failure in RL-trained reasoning

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Dr. MAMR (Multi-Agent Meta-Reasoning Done Right) diagnoses "lazy agent" collapse in multi-agent LLM reasoning—where one agent dominates and the other becomes trivial—and fixes it via three coordinated mechanisms: a corrected GRPO normalization, Shapley-inspired causal influence scoring, and a verifiable restart reward that lets the reasoning agent discard noisy intermediate context.

Problem

Multi-agent LLM reasoning frameworks (specifically ReMA) suffer from lazy agent behavior: the reasoning agent progressively stops contributing substantively, copying or summarizing the meta-thinking agent rather than reasoning independently. This collapses the two-agent system into an effective single agent. The root cause is a structural bias in multi-turn GRPO's \(\frac{1}{T_i}\) normalization term, which penalizes longer trajectories and implicitly incentivizes the model to minimize the number of turns—rewarding lazy shortcuts.

Method

Dr. MAMR addresses the problem in three stages:

  1. Remove \(\frac{1}{T_i}\) normalization: Eliminates the turn-count bias from multi-turn GRPO's objective (Eq. 2), partially reducing lazy behavior.

  2. Shapley-inspired causal influence (CI): Each step \(s_{i,t}\) is assigned an influence score by grouping semantically similar steps across rollouts into \(G_S(s_{i,t})\), measuring each group member's one-step causal influence via attention suppression (\(\Delta\ell_{j,t'} = \log p_\text{mask} - \log p_\text{full}\)), and averaging: $\(\text{CI}(s_{i,t}) = \frac{1}{|G_S(s_{i,t})|} \sum_{s_{j,t'} \in G_S(s_{i,t})} \Delta\ell_{j,t'}\)$ This avoids expensive resampling while reducing phrasing bias.

  3. Verifiable restart reward: A <restart> control token lets the reasoning agent discard its prior outputs and re-aggregate instructions. The reward is binary: +1 if masking prior reasoning increases confidence in the correct final answer (\(z_i=+1, \Delta\ell_{i,t}>0\)), −1 otherwise. The aggregated step-level advantage is: $\(A^\text{step}_{i,t} = \tilde{A}_{i,t} + \alpha \tilde{C}_{i,t} + \beta \tilde{R}_{i,t}\)$ where \(\tilde{A}\) is outcome-based, \(\tilde{C}\) is causal influence, and \(\tilde{R}\) is the restart signal.

Key Contributions

  • Empirical and theoretical identification of lazy agent behavior in sequential multi-agent LLM systems, with a formal proof (Theorem 1) that the \(\frac{1}{T_i}\) turn normalization in multi-turn GRPO biases training toward shorter trajectories.
  • Shapley-inspired causal influence estimator that is tractable during online RL by grouping semantically equivalent steps across rollouts.
  • Verifiable restart reward mechanism enabling the reasoning agent to recover from error propagation in extended multi-turn interactions.
  • Aggregated step-level advantage function combining outcome reward, causal influence, and restart signals.

Results

  • MATH500: Dr. MAMR achieves 78.4 vs. 74.4 (ReMA) and 75.0 (untrained base), a +4.0-point gain over ReMA.
  • AMC23 and Olympiad: ~8% Pass@1 gain for ReMA+ (deliberation at inference only) over ReMA.
  • AIME24/AIME25: ~7% Pass@16 gain from deliberation.
  • ReMA+ (prompt-only deliberation) outperforms ReMA on 6 of 8 benchmarks under Pass@16.
  • Causal influence distributions confirm reasoning agent engagement recovers to near-initialization levels under Dr. MAMR (Fig. 2d vs. 2b).

Limitations

  • Experiments are confined to mathematical reasoning (DeepScaleR training set, MATH500/AIME/AMC/GSM8K benchmarks); generalization to other domains is unverified.
  • Semantic grouping for Shapley-inspired CI requires a similarity measure whose sensitivity is not fully ablated.
  • Hyperparameters \(\alpha\) and \(\beta\) in the advantage aggregation require tuning.
  • The restart mechanism assumes the model can reliably identify when prior outputs are noisy—this may fail in early training stages.
  • Both agents share weights (distinguished only by system prompts), which may limit the degree of specialization achievable.

Relevance to Harnesses / Meta-Harnesses

Dr. MAMR is directly a meta-harness paper: it studies a harness in which a meta-thinking agent orchestrates a reasoning agent across sequential turns, and systematically diagnoses failure modes that emerge from the harness's training objective rather than from either individual agent. The theoretical result that the GRPO normalization term structurally degrades multi-agent collaboration is a foundational insight for anyone designing RL-trained orchestration harnesses. The verifiable restart reward and Shapley-inspired credit assignment are reusable harness-level mechanisms applicable to any multi-agent pipeline where intermediate context can mislead downstream agents—a common failure mode in complex orchestration architectures.