Skip to content

CoMAS: Co-Evolving Multi-Agent Systems via Interaction Rewards

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Interaction-reward-driven co-evolution of specialized agents; novel self-improvement paradigm

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CoMAS (Co-Evolving Multi-Agent Systems) is an RL-based framework that trains LLM agents to self-improve by deriving reward signals purely from inter-agent discussion dynamics—no external verifiers or reward models required. Agents alternate between proposing solutions, critically evaluating peers, and scoring those evaluations; the resulting zero-sum reward signal is used with REINFORCE++ to update each agent's policy. The framework achieves state-of-the-art performance across math, coding, science, and general-knowledge benchmarks in both single-agent and multi-agent inference setups.

Problem

Existing self-evolving LLM agents either rely on dense external rewards (rule-based verifiers, specialized reward models) or intrinsic self-rewarding signals (self-certainty, confidence, semantic entropy, majority-voting pseudo-labels). Both paradigms diverge from how human intelligence evolves—through collective, multi-individual discussion—and the self-rewarding approaches are prone to reward hacking and training collapse. There is no prior work that derives training signal purely from structured inter-agent interaction without any external oracle.

Method

CoMAS structures training around a three-step interaction loop per question:

  1. Interaction — \(l\) heterogeneous or homogeneous agents participate in \(m\) rounds. Each round: one agent (sampled uniformly) proposes a solution \(s_i\) conditioned on the question \(q\) and compressed discussion history \(h_q\) (last \(\kappa\) rounds); \(n\) agents each produce a critical evaluation \(e_{i,j}\) of \(s_i\); a scoring step produces \(\tau_{i,j} \in \{1,2,3\}\) assessing the solution quality given the evaluation (via LLM-as-a-judge).

  2. Reward formulation — Scores are normalized and converted to complementary (zero-sum) rewards: $\(r(s_i) = \frac{\hat{\tau}_{i,j} - 1}{2}, \quad r(e_{i,j}) = 1 - r(s_i)\)$ A format-violation penalty \(r(\tau_{i,j}) = -1\) enforces scorer compliance. This creates a zero-sum game between solver and evaluator, simultaneously rewarding correctness and critical thinking.

  3. Policy optimization — Each agent's replay buffer \(D_k\) accumulates \((p, o, r(o))\) triples across all three interaction roles. REINFORCE++ with a token-level KL-penalized advantage: $\(A_t = r(o) - \beta \sum_{\lambda=t}^{|o|} \log \frac{\pi_{\theta_k}(o_\lambda | p, o_{<\lambda})}{\pi_{\text{ref}}(o_\lambda | p, o_{<\lambda})}\)$ is used to update each agent's weights independently, enabling decentralized co-evolution.

Main configuration: Qwen2.5-3B-Instruct, \(l=4\) agents, \(m=8\) rounds, \(n=1\) evaluation per solution, \(\kappa=2\) round horizon; training on 2000 samples from MATH, KodCode, and WebInstruct-verified.

Key Contributions

  • Novel interaction reward paradigm: training signal derived entirely from structured peer discussion (solution → evaluation → scoring), with no external verifier or reward model.
  • A zero-sum reward formulation that simultaneously incentivizes correct solving and rigorous critical evaluation, preventing catering bias.
  • Support for heterogeneous agent systems—different foundation models can co-evolve within the same framework without a shared backbone.
  • Demonstration that interaction-based rewards prevent training collapse and reward hacking (unlike TTRL and MAPoRL in certain setups).
  • Empirical evidence of scalability: performance improves as the number and diversity of agents increase.

Results

All gains reported relative to untrained Qwen2.5-3B-Instruct baseline:

  • Vanilla (single-agent): +1.40% GSM8K (85.40%), +1.83% HumanEval (70.73%), +2.00% SciBench (34.67%), +1.00% MMLU (62.40%); competitive on remaining benchmarks.
  • Consistency (single-agent): +3.66% HumanEval (77.44%), +3.40% MBPP (59.20%), +2.40% MMLU (65.60%).
  • AutoGen (multi-agent): +19.80% GSM8K (72.40%), +10.98% HumanEval (50.61%), +13.20% MMLU (50.60%); TTRL collapses (−11.60% GSM8K), MAPoRL regresses on several benchmarks.
  • Debate (multi-agent): +6.10% HumanEval (77.44%), best/near-best on MATH-500 (55.40%) and MMLU (65.20%).
  • CoMAS on Qwen2.5-7B-Instruct validated in appendix (general effectiveness across model scales).
  • Training dynamics show stable reward increase and response-length growth for all agents (Figure 3).

Baselines compared: untrained agent, SRLM (DPO self-rewarding), MAPoRL (multi-agent RL with verifier), TTRL (majority-vote pseudo-labels).

Limitations

  • Evaluated primarily on Qwen2.5-3B-Instruct; appendix extends to 7B but larger frontier models are not tested.
  • Training data is limited to 2000 samples across three narrow domains (math, coding, science); breadth of training distribution is constrained.
  • The LLM-as-a-judge scoring step is itself performed by the same class of agents being trained, which may introduce circular biases not fully analyzed.
  • Heterogeneous co-evolution results are presented qualitatively; quantitative gains from mixing diverse backbone models are not systematically isolated.
  • Interaction cost scales with \(l \times m \times n\); training compute analysis is deferred to an appendix and not integrated into the main comparison.
  • GPQA results show only modest or inconsistent gains, suggesting limits on tasks requiring deep domain expertise beyond the training distribution.

Relevance to Agentic AI / LLM Agents

CoMAS directly advances the self-evolving agent paradigm by replacing the need for external reward infrastructure with a structured multi-agent social signal—an important step toward deployable autonomous agents that can bootstrap capability without human-curated reward labels. The decentralized, heterogeneous co-evolution mechanism is directly applicable to practical multi-agent pipelines (AutoGen, debate frameworks) and addresses a known failure mode where individual RL methods degrade in collaborative inference settings. The zero-sum solver–evaluator dynamic is a generalizable design pattern for any agentic system that pairs a generator with a critic. For researchers tracking LLM agents, CoMAS provides a concrete, reproducible recipe showing that inter-agent interaction alone is a sufficient training signal—opening the door to continual self-improvement in deployed multi-agent systems without reward model maintenance.