Skip to content

Unlocking the Power of Multi-Agent LLM for Reasoning: From Lazy Agents to Deliberation

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Multi-agent RL with meta-thinking agents improves complex reasoning

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Multi-agent LLM reasoning frameworks suffer from "lazy agent" behavior where one agent dominates while the other contributes trivially, collapsing collaboration into an effective single agent. This paper identifies the root cause in a structural bias in multi-turn GRPO's normalization term, then proposes Dr. MAMR: a framework combining Shapley-inspired causal influence estimation and a verifiable restart reward to enforce balanced agent contributions and deliberative recovery from noisy multi-turn context.

Problem

ReMA-style multi-agent reasoning frameworks (meta-thinking agent + reasoning agent trained with multi-turn GRPO) suffer from lazy agent behavior: the reasoning agent learns to summarize or copy the meta-thinking agent's outputs rather than genuinely contribute, causing the reasoning agent's causal influence to collapse during training. This undermines the core premise of multi-agent collaboration and can actually hurt performance relative to baseline (74.4 vs. 75.0 on MATH500 after training with ReMA). Additionally, as agents engage more, longer multi-turn interactions cause error propagation and context confusion.

Method

Dr. MAMR (Multi-Agent Meta-Reasoning Done Right) addresses the lazy agent problem through three components:

  1. Theoretical fix to multi-turn GRPO: Removes the \(\frac{1}{T_i}\) turn-normalization factor from the GRPO objective (Eq. 2). The paper proves (Theorem 1) that this normalization biases optimization toward shorter trajectories: if a short trajectory \(\tau^S\) (horizon \(T^S\)) and long trajectory \(\tau^L\) (horizon \(T^L > T^S\)) have equal final reward, the gradient update favors \(\tau^S\) unless \(\kappa = \frac{\|Z_t(\tau^L)\|}{\|Z_t(\tau^S)\|} \geq \frac{T^L}{T^S}\), incentivizing lazy (turn-reducing) behavior.

  2. Shapley-inspired causal influence (CI): For each step \(s_{i,t}\) in a flattened trajectory, groups semantically similar steps \(\mathcal{G}_S(s_{i,t})\) across rollouts and computes per-step causal influence as the average log-probability difference of the next step under full vs. masked history: $\(\text{CI}(s_{i,t}) = \frac{1}{|\mathcal{G}_S(s_{i,t})|} \sum_{(j,t'): s_{j,t'} \in \mathcal{G}_S(s_{i,t})} \Delta\ell_{j,t'}\)$ where \(\Delta\ell_{j,t'} = \log p_{\text{mask}} - \log p_{\text{full}}\) (Eq. 4–5). Grouping avoids expensive resampling during online RL and reduces phrasing bias.

  3. Verifiable restart reward: Introduces a <restart> control token allowing the reasoning agent to discard prior outputs and re-aggregate instructions. A binary restart reward \(r^{\text{restart}}_{i,t}\) is assigned based on whether masking previous reasoning outputs changes the model's log-probability of generating the correct final answer in the correct direction (Eq. 6–7). The final step-level advantage aggregates outcome-based advantage, CI, and restart signals: $\(A^{\text{step}}_{i,t} = \tilde{A}_{i,t} + \alpha \tilde{C}_{i,t} + \beta \tilde{R}_{i,t}\)$

Key Contributions

  • Empirical identification and theoretical explanation of lazy agent behavior in multi-turn GRPO: the \(\frac{1}{T}\) normalization creates a gradient bias toward fewer-turn trajectories, preferentially reinforcing lazy (turn-minimizing) behavior.
  • Shapley-inspired step-level causal influence estimation that operates without additional sampling by grouping semantically similar steps across concurrent rollouts.
  • A novel verifiable reward for <restart> behavior that provides a grounded signal (based on log-probability shift under history masking) for whether discarding prior context improves correctness probability.
  • Aggregated step-level advantage combining outcome reward, CI, and restart signal into a single training objective.

Results

  • Dr. MAMR achieves 78.4 on MATH500, compared to 74.4 (ReMA), 75.6 (ReMA + prompt), and 75.0 (untrained baseline).
  • ReMA+ (inference-time restart prompting only, no training) outperforms standard ReMA by ~8% on AMC23 and Olympiad (Pass@1) and ~7% on AIME24/AIME25 (Pass@16), validating the deliberation hypothesis before training.
  • ReMA+ outperforms ReMA on 6 of 8 benchmarks under Pass@16, with gains widening on harder benchmarks and longer interactions.
  • Ablation shows removing \(\frac{1}{T}\) normalization alone alleviates but does not eliminate lazy behavior; CI and restart components provide additional gains.
  • Training on DeepScaleR dataset; evaluation on MATH500, GSM8K, AIME24, AIME25, AMC23, Gaokao2023en, Minerva Math, and Olympiad benchmarks.

Limitations

  • Experiments are restricted to mathematical reasoning tasks; generalizability to other domains (code, planning, tool use) is not demonstrated.
  • Semantic similarity grouping for CI requires a similarity measure that may introduce its own hyperparameter sensitivity; practical details are deferred to an appendix.
  • The <restart> mechanism adds inference-time overhead (masking and re-aggregation); cost is not quantified relative to baseline.
  • Shapley-inspired CI is an approximation; full Shapley computation is acknowledged as infeasible, so the quality guarantee of the approximation is informal.
  • The paper focuses on a two-agent (meta-thinking + reasoning) setup; scaling to \(k > 2\) specialized agents is unaddressed.

Relevance to Agentic AI / LLM Agents

This paper directly addresses a fundamental failure mode in trained multi-agent LLM systems: role collapse under RL, where the multi-agent structure offers no practical benefit over a single agent. The theoretical analysis of turn-normalization bias in GRPO provides an actionable explanation for why RL-trained agent systems degrade, which is broadly applicable to any multi-turn MARL training pipeline. The Shapley-inspired CI measure and verifiable restart reward are practically deployable components for improving credit assignment in sequential LLM agent systems, advancing the goal of building agents that collaboratively decompose and solve hard problems without collapsing to degenerate interaction patterns.