Skip to content

MARTI: A Framework for Multi-Agent LLM Systems Reinforced Training and Inference

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Open-source framework for scalable multi-agent LLM training and inference

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MARTI is an open-source framework that bridges multi-agent LLM inference and distributed RL training — a gap no prior framework addressed. It enables reinforcement learning directly over multi-agent workflows (debate, mixture-of-agents, chain-of-agents) with centralized reward allocation and distributed per-agent policy updates. Experiments on competition math show multi-agent RL consistently exceeds single-agent RL under the same inference budget after convergence.

Problem

Existing LLM multi-agent frameworks (AutoGen, CAMEL, MetaGPT) rely purely on inference and cannot train agents via RL; existing RL frameworks (OpenRLHF, veRL, TRL) train single agents and have no multi-agent interaction support. This leaves a gap: multi-agent systems underperform majority voting because agents are trained on single-agent objectives and fail to leverage inter-agent information. There is no unified framework supporting both multi-agent rollouts and RL policy training.

Method

MARTI is built on OpenRLHF and consists of three modules:

  1. Multi-Agent World: executes prompt-driven rollouts via pluggable workflow interfaces (Multi-Agent Debate / MAD, Mixture-of-Agents / MoA, Chain-of-Agents / CoA); supports asynchronous generation to avoid batch-synchronization bottlenecks across agents with unequal generation lengths.

  2. Centralized Reward Models: after world interactions, computes global rewards using either rule-based verifiers (DeepSeek-R1-style) or LLM-based generative reward models (GenRMs), then decomposes them per agent via credit assignment and influence-aware reward shaping adapted from MAPoRL. The shaped reward is: $\(\tilde{R}_t^i = R_t^i + \alpha \cdot \Delta_t^i\)$ where \(\Delta_t^i\) is either Margin Mode (\(R_t^i - Q_t^i\)) or Quality Mode (\(Q_t^i R_t^i - (1-Q_t^i)(1-R_t^i)\)), with \(Q_t^i\) being the agent's historical average reward over scope \(H_t^i\).

  3. Agent Policy Trainer: distributes per-agent trajectories + shaped rewards to individual policy trainers running REINFORCE++, GRPO, or PPO (and optionally SFT/DPO for stability). All agents share the same RL algorithm.

Reasoning models use Test-Time RL (TTRL) fine-tuning on AIME data; base models use zero-like RL on MATH Level 3–5.

Key Contributions

  • First framework unifying multi-agent LLM inference and distributed RL policy training (no prior framework supported both).
  • Multi-turn asynchronous rollout support for multi-agent scenarios, improving throughput over synchronous batching.
  • Influence-aware reward shaping with two modes (Margin/Quality) and two historical scopes, decomposing global rewards into per-agent training signals.
  • Support for both rule-based and LLM-based (GenRM) reward models within the same pipeline.
  • Open-source release enabling reproducible multi-agent RL research.

Results

  • DeepScaleR-1.5B-Preview MAD (MARTI, TTRL): 66.7 on AIME24 — outperforms OpenAI-o1-mini and single-agent TTRL baseline.
  • Qwen2.5-3B MAD 2×2 (MARTI, GRPO): avg 46.0 vs. single-agent GRPO 37.9 and single-agent Maj@4 RL 30.0 across AIME/AMC/MATH-500.
  • Llama-3.2-3B-Instruct MAD 2×2 (MARTI): avg 32.1 vs. single-agent RL 28.7 and Maj@4 RL 30.0.
  • Reward shaping ablation (Qwen2.5-3B MAD): removing shaping drops avg from 45.6 → 36.6; MoA drops from 43.1 → 38.1.
  • Asynchronous rollout speedup: at 512 concurrent workers, Chain-of-Agents rollout time falls from 612.6 s (sync) to 498.4 s; gains grow with interaction rounds (e.g., 8-round MAD: 3308 s sync → ~2928 s async×64).
  • Untrained multi-agent workflows underperform majority voting at equal compute; MARTI training closes and exceeds this gap.

Limitations

  • Experiments limited to competition-level mathematics; real-world applicability and open-domain transfer not demonstrated.
  • Multi-agent RL introduces substantially higher compute cost: each agent requires a dedicated GPU node (3 nodes × 8 A800 80GB for 3 agents).
  • Credit assignment and reward decomposition for MAS remain unsolved in general; GenRM-based rewards for multi-agent roles are flagged as future work.
  • Base model pass@k ceiling still constrains the RL upper bound; MARTI cannot overcome pre-training limitations.
  • Asynchronous speedup is moderate and concurrency-dependent; gains diminish at lower concurrency levels.

Relevance to Agentic AI / LLM Agents

MARTI directly addresses a core bottleneck in agentic LLM research: the inability to train multi-agent systems end-to-end with RL, not just prompt them at inference time. The finding that inference-only multi-agent workflows underperform majority voting — but RL-trained ones exceed it — is a strong empirical signal that agentic collaboration requires dedicated training objectives, not just better prompting. The credit-assignment and reward-shaping machinery generalizes beyond math to any multi-agent setting with verifiable rewards, making this a foundational infrastructure contribution for the field. For researchers tracking agentic RL (alongside DeepSeek-R1, GRPO, agentic RLHF), MARTI provides the missing training loop for multi-agent reasoning systems.