Skip to content

SiriuS: Self-improving Multi-agent Systems via Bootstrapped Reasoning

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SiriuS is a self-improving optimization framework for multi-agent LLM systems that builds an experience library of successful reasoning trajectories via bootstrapped fine-tuning, augmenting failed trajectories with grounded feedback before discarding them. It improves accuracy 2.86–21.88% over baselines on reasoning and biomedical QA, and strengthens competitive negotiation agents—all without human-annotated intermediate supervision.

Problem

Multi-agent LLM systems rely on fragile, hand-crafted prompts and heuristics. Optimizing them is hard because (1) task-level reward signals exist but credit assignment across individual agents is ambiguous, and (2) there is no principled way to generate role-specific training data for specialized agents within a collaborative pipeline.

Method

SiriuS iterates over T fine-tuning rounds. Each round: 1. Action sampling: each agent A⁽ⁿ⁾ generates responses conditioned on prior agents' outputs (via a topological directed graph). 2. Trajectory evaluation: trajectories are scored by a payoff function Rᵢ; those exceeding a reward threshold ε are added to agent-specific "good trajectory sets." 3. Trajectory augmentation: failed trajectories are refined by prompting the agent with feedback generated by an auxiliary agent grounded in the ground truth, then rephrasing the corrected output into a direct problem-solving step—expanding the training pool. 4. Supervised fine-tuning (SFT): each agent is fine-tuned separately on its own good trajectory set.

Three multi-agent configurations are studied: (a) sequential domain-expert pipelines (problem solving), (b) Actor-Judgment-Critic loops, and (c) two-player competitive negotiation games.

Key Contributions

  • Experience library: a growing repository of high-quality, agent-specific reasoning trajectories collected from successful multi-agent interactions.
  • Trajectory augmentation pipeline: converts failed trajectories into usable training data via feedback-guided regeneration, without requiring direct ground-truth labels per intermediate step.
  • Joint, role-specialized fine-tuning: demonstrates that separate per-role SFT outperforms fine-tuning a single shared model on combined data.
  • Competitive-setting optimization: extends bootstrapped self-play to multi-turn negotiation games (Resource Exchange, Ultimatum, Seller-Buyer).
  • Scalable self-improvement: iterative rounds yield marginal additional gains, and learned strategies generalize to unseen resource configurations.

Results

  • College Physics (GPT-3.5): SiriuS 33.96% vs. best baseline TextGrad 32.09% (+1.87 pp) and single-agent 25.55%.
  • College Chemistry (GPT-3.5): SiriuS 55.90% vs. best baseline CoMM 47.69% (+8.21 pp); vs. single-agent 40.00% (+15.9 pp).
  • PubMedQA (GPT-3.5): SiriuS 75.33% vs. CoMM 72.27% (+3.06 pp); vs. single-agent 57.53% (+17.8 pp).
  • PubMedQA (GPT-4o-mini): SiriuS 73.67% vs. CoMM 70.27% (+3.4 pp).
  • PubMedQA (Llama-3.2-3B): SiriuS 68.27% vs. CoMM 65.80% (+2.47 pp).
  • Actor-Critic on PubMedQA (GPT-4o-mini): SiriuS 66.80% overall accuracy vs. Prompt 58.20% and Self-Correct 28.80%; TP accuracy 59.80% vs. Prompt 51.60%.
  • Competitive settings: SiriuS consistently improves win rate and payoff in Resource Exchange and Ultimatum; seller-side price improves from below-50 to ~50 in Buyer-Seller.
  • Ablation: removing augmentation drops PubMedQA from 75.33% to 73.40% (GPT-3.5); mixing SiriuS with base agents and using a single shared fine-tuned model both degrade performance.

Limitations

  • Evaluated only on GPT-3.5-turbo, GPT-4o-mini, and Llama-3.2-3B; reliance on OpenAI's fine-tuning API limits reproducibility and generality.
  • Trajectory augmentation requires an auxiliary agent with access to ground truth for feedback generation, which may not be available in all tasks.
  • Credit assignment is still implicit (whole-trajectory retention) rather than step-level; the approach sidesteps rather than solves the attribution problem.
  • Iterative fine-tuning yields only marginal gains beyond a single round, suggesting limited scalability with additional iterations.
  • Competitive setting experiments use fixed, relatively simple negotiation games; transfer to open-ended agentic tasks is not demonstrated.
  • No theoretical analysis of convergence or diversity of the experience library.

Relevance to Agentic AI / LLM Agents

SiriuS directly addresses one of the core open problems in multi-agent LLM systems: how to generate training signal for role-specialized agents when only outcome-level rewards are available. By framing this as bootstrapped SFT over an experience library—analogous to STaR for single agents—it offers a principled, scalable alternative to prompt engineering and gradient-based prompt optimization (TextGrad, DSPy). The trajectory augmentation mechanism is particularly relevant to researchers building self-correcting or self-play agent pipelines, as it converts failure data into usable supervision without human labeling. The Actor-Judgment-Critic experiments also expose a concrete failure mode of untuned self-correction (correct solutions being inadvertently modified), directly informing design choices for verification-aware agentic loops.