ReMA: Learning to Meta-Think for LLMs with Multi-agent Reinforcement Learning¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

ReMA (Reinforced Meta-thinking Agents) frames meta-thinking as a two-agent MARL problem: a high-level agent generates strategic plans/oversight, and a low-level agent executes detailed reasoning. Trained end-to-end with REINFORCE++/GRPO, ReMA consistently outperforms single-agent RL baselines on competitive math and LLM-as-a-Judge benchmarks, particularly on out-of-distribution tasks.

Problem¶

Single-agent RL approaches to eliciting meta-thinking (monitoring, planning, self-correction) in LLMs face inefficient exploration because one model must simultaneously learn both strategic oversight and detailed reasoning in a single autoregressive pass. Supervised construction-based methods (SFT on predefined templates) fail to generalize OOD. Single-agent RL either requires very strong base models or converges to local optima with poor readability.

Method¶

ReMA defines a Multi-Agent Meta-thinking Reasoning Process (MAMRP) by decoupling reasoning into two hierarchical LLM agents: πh (high-level, meta-thinking) and πl (low-level, reasoning). In the single-turn formulation, πh produces a complete meta-thinking trace (plans, decompositions, verification signals) and πl executes the solution given that trace. Both agents are trained with aligned outcome rewards via iterative alternating RL (freeze one, update the other).

For multi-turn extension (T > 1 turns), the agents interleave: πh → πl → πh → ... → final answer. To handle this efficiently, the authors adopt parameter sharing (both agents share weights θ, distinguished only by system prompts Sh/Sl) and introduce a turn-level ratio for GRPO clipping instead of token-level ratio. The turn-level ratio treats all tokens within a single agent turn as one action unit, avoiding length bias and training instability (repetition collapse, empty-response degeneration).

Key Contributions¶

First formal definition and RL optimization of Multi-Agent Meta-thinking Reasoning Process (MAMRP) via MARL.
Hierarchical two-agent decomposition that separates meta-thinking from reasoning, improving exploration efficiency over single-agent RL.
Turn-level ratio extension to GRPO for stable multi-turn MARL training, with trajectory-level loss aggregation.
Parameter-sharing strategy for multi-turn settings (shared weights + role-specific system prompts) to eliminate GPU model-swap overhead and enable larger batch sizes.
Empirical evidence that meta-thinking boosts OOD generalization, with ablation revealing emergent behaviors (role reversal, scale-dependent action diversity).

Results¶

Math reasoning (greedy decoding, pass@1 vs. VRP/VRPRL/MRPRL): - Llama3-8B-Instruct: ReMA average +6.68% over CoT; AMC23 +20.00% (largest OOD gain) - Llama3.1-8B-Instruct: ReMA average +1.08%; AIME24 +3.33%, Minerva Math +5.52% - Qwen2.5-7B-Instruct: ReMA average +2.85%; AIME24 +13.33%, AMC23 +10.00%

LLM-as-a-Judge: - Llama3.1-8B: RewardBench970 +14.23% over CoT, average +7.92% - Qwen2.5-7B: average +1.75% (MRPRL slightly higher on this model)

Multi-turn ReMA (Qwen2.5-7B, MATH-Level3-5): - Training accuracy improves from ~55% to ~70%; ~5% average gain across 7 OOD test benchmarks - Turn-level ratio reaches higher reward with fewer steps than token-level ratio - Shared parameters converge noticeably faster than separate parameters

Meta-learning ablation (Qwen2.5-Math-7B base): RL under meta-thinking achieves best generalization on AIME24; SFT-then-RL excels on easier in-distribution problems but saturates on hard problems.

Limitations¶

Multi-turn ReMA is highly sensitive to hyperparameters (max tokens per turn, max turns); certain configs cause repetition collapse or early empty-response termination ("Echo Trap").
Multi-turn RL faces long-horizon credit assignment and state drift; requires careful tuning of training recipe (model, data, algorithm, RL infra).
Gains from RL on instruction-tuned models are smaller than from base models due to restricted output distribution.
Smaller LMs (1B) cannot maintain structured meta-thinking outputs (collapse to EMPTY action), limiting applicability.
Multi-turn bootstrapping requires a small supervised dataset constructed with GPT-4o, adding a data dependency.
Nondeterminism between inference and training engines (noted as an open infra problem) can destabilize training.

Relevance to Agentic AI / LLM Agents¶

ReMA is a direct instantiation of multi-agent collaboration as a training-time mechanism rather than just an inference-time scaffold — the hierarchical planner/executor split mirrors standard agentic architectures (task manager + worker agent) but optimizes the division of cognitive labor via RL rather than hand-engineering. The MAMRP formulation provides a principled framework for thinking about how to decompose long-horizon reasoning into roles with aligned rewards, directly relevant to building LLM agents that plan, self-monitor, and adapt. The turn-level ratio and parameter-sharing findings are practically important for anyone training multi-turn agentic systems. The observed "role reversal" and scale-dependent meta-thinking behaviors are qualitative insights into emergent agent specialization under MARL.