Multi-Agent Design: Optimizing Agents with Better Prompts and Topologies

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Hamid Palangi (Microsoft); optimizing multi-agent prompts and topologies jointly

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MASS (Multi-Agent System Search) is a three-stage automated framework for jointly optimizing both prompts and topologies in multi-agent LLM systems. It reveals that prompt quality is more influential than topology scaling, and that only a small fraction of topologies are beneficial. MASS achieves state-of-the-art results across reasoning, multi-hop QA, and code generation benchmarks.

Problem

Designing effective multi-agent systems (MAS) requires jointly optimizing prompts and topologies, but prior work treats these in isolation: topology search methods (ADAS, AFlow) use hand-crafted prompts, while prompt optimizers (DSPy/MIPRO) ignore topology interactions. Naive joint optimization is intractable due to (1) prompt sensitivity compounding across cascaded agents, (2) sparse reward signals with growing \(n\) agents, and (3) exponential combinatorial search over topology configurations.

Method

MASS formalizes MAS design as \(W^*(a) = \arg\max_{a \sim \mathcal{A}} \mathbb{E}_{(x,y)\sim D}[f(W(a(x)), y)]\) and solves it via three interleaved stages:

1. Block-level prompt optimization (1PO): Runs APO (MIPRO: joint instruction + 1-shot exemplar optimization) on each topology building block independently with minimal agent count. Measures incremental influence \(I_{a_i} = E(a_i^*)/E(a_0^*)\) per block.

2. Workflow topology optimization (2TO): Prunes the search space via influence-weighted rejection sampling: \(p_a = \text{Softmax}(I_a, t)\); block \(a_i\) is rejected if \(u \sim \text{Uniform}(0,1) > p_{a_i}\). Valid configurations are sampled within a maximum agent budget \(B\), assembled in a fixed rule-based order [summarize → reflect → debate → aggregate], then evaluated on a validation split.

3. Workflow-level prompt optimization (3PO): Re-runs APO over all agents jointly conditioned on the best topology from Stage 2, adapting individual block prompts to their orchestration context.

The configurable building blocks are: Aggregate (parallel agents + majority vote, parameterized by \(N_a\)), Reflect (\(N_r\) self-refinement rounds), Debate (\(N_d\) fully-connected rounds), Summarize (\(N_s\) abstraction rounds for long-context), and Tool-use (binary insertion \(N_T \in \{0,1\}\)).

Key Contributions

• Empirical analysis showing prompt optimization is more token-efficient than topology scaling (self-consistency, reflect, debate) on MATH with Gemini 1.5 Pro.
• Demonstration that beneficial topologies are a small fraction of the full search space (e.g., only debate yields +3% on HotpotQA; summarize and reflect hurt performance).
• MASS framework: plug-and-play with any prompt optimizer; three-stage local-to-global optimization that avoids joint combinatorial search.
• Design principles for effective MAS derived from the strongest MASS-found systems.

Results

• Gemini 1.5 Pro: MASS achieves 78.79% average across 8 tasks vs. best baseline Multi-Agent Debate at 70.26% — an improvement of ~8.5 points average.
• MATH: 84.67% vs. ADAS 80.00%
• DROP: 90.52% vs. AFlow* 88.92%
• HotpotQA: 69.91% vs. AFlow* 68.62%
• MuSiQue: 51.40% vs. Multi-Agent Debate 46.00%
• 2WikiMQA: 73.34% vs. AFlow 76.51% (AFlow uses Claude 3.5 Sonnet as optimizer)
• MBPP: 86.50% vs. ADAS 73.00%
• HumanEval: 91.67% vs. AFlow* 88.00%
• LCB: 82.33% vs. Multi-Agent Debate 73.67%
• Gemini 1.5 Flash: MASS achieves 74.30% average vs. best baseline Multi-Agent Debate at 65.91%.
• All results reported as mean ± std over 3 runs; inference cost controlled comparably across baselines (max 10 agents per query, see Table 7).
• Key findings confirmed on Claude 3.5 Sonnet and Mistral Nemo (Appendix D).

Limitations

• AFlow comparison is partial: AFlow's meta-prompt works only with Claude 3.5 Sonnet as optimizer; Gemini-executor + Claude-optimizer results are flagged as reference-only, making direct comparison imperfect.
• Stage 1 influence measurement uses a held-out validation set, so influence estimates may not generalize across domains without re-running block-level optimization.
• The rule-based assembly order [summarize → reflect → debate → aggregate] is fixed by design, potentially excluding beneficial non-standard orderings.
• Experiments use Gemini 1.5 family as primary models; generalization to other frontier models is only partially validated.
• Search budget \(B\) (max agents) caps topology expressiveness; very large MAS (>10 agents) are not evaluated.

Relevance to Agentic AI / LLM Agents

MASS directly advances the science of automated MAS design by quantifying for the first time the relative importance of prompt quality vs. topology structure — a foundational question for anyone building multi-agent pipelines. The result that prompt optimization dominates topology scaling in token efficiency challenges the prevailing assumption that topology engineering is the primary lever for MAS performance. The influence-weighted search space pruning offers a practical, general-purpose recipe for constraining the combinatorial explosion inherent in MAS design, applicable beyond the specific building blocks tested here. Together with the empirical finding that only a small subset of topologies are beneficial, MASS provides actionable design principles (optimize locally before scaling globally) that should inform future agentic frameworks.