Skip to content

When Do Multi-Agent Systems Help? An Information Bottleneck Perspective

๐Ÿ•’ Published (v1): 2026-07-17 17:13 UTC ยท Source: Arxiv ยท link

Why this paper was selected

Information-bottleneck theory for when MAS beats SAS; directly governs single- vs multi-agent architecture decisions

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

๐Ÿ’ฌ Ask ChatGPTโœฆ Ask Claude

TL;DR

This paper explains when LLM-based multi-agent systems (MAS) outperform single-agent systems (SAS) using the information bottleneck (IB) principle. The key insight is that MAS gains and losses arise entirely from bounded relay compression between agents, not from decomposition alone. MAS helps when upstream context reduction outweighs relay information loss, with this balance shifting predictably with model capability.

Problem

Empirical results on MAS vs. SAS are inconsistent: MAS gains vary across tasks and shrink as base model capability increases, but no mechanistic explanation exists for why MAS helps in some settings and hurts in others. Existing comparisons are purely empirical and conflate the effects of task decomposition with those of inter-agent communication.

Method

The paper formalizes SAS and MAS under a common agent abstraction, then proves an infinite-bandwidth equivalence: any SAS can be simulated by a MAS that transmits the full upstream context, so nontrivial MAS effects arise only from bounded relay compression.

Each inter-worker relay \(m_i\) (compressing upstream context \(M_i\) for downstream worker \(i+1\)) is modeled as an information bottleneck with objective:

\[\min_{p(m_i|M_i)} I(M_i; m_i) - \beta I(m_i; Y_{i+1} | X_{i+1})\]

The effective parameter \(\beta\) is interpreted as a proxy for downstream LLM capability (stronger models โ†’ larger \(\beta\), valuing preserved predictive information more). The MAS gain at each relay stage decomposes as:

\[G_i^{\text{MAS}} = \underbrace{H(M_i | m_i)}_{\text{upstream context reduction}} - \underbrace{\frac{\Delta_i(m_i)}{\beta}}_{\text{relay info loss, capability-weighted}}\]

where \(\Delta_i(m_i) \triangleq I(M_i; Y_{i+1}|X_{i+1}) - I(m_i; Y_{i+1}|X_{i+1}) \geq 0\) is the relay information loss.

Three controlled prototypes isolate relay effects: SAS (full shared context), SAS-contextflow (same planner decomposition, full context propagated), and MAS (same decomposition, but workers communicate only via compressed relays). Relay complexity \(\delta\) โ€” the minimum description length of a downstream-sufficient relay โ€” is used to classify benchmarks into regimes (\(\delta \approx 0\), \(\delta > 0\), \(\delta \gg 0\)).

Key Contributions

  • MASโ€“SAS equivalence theorem: under infinite relay bandwidth, any SAS is exactly simulable by a MAS, proving that relay compression is the sole source of MAS variability.
  • IB-based MAS gain decomposition (Theorem 4.1): closed-form stage-wise gain as a trade-off between context reduction and capability-weighted relay information loss.
  • Capability-dependent ฮฒ: formalizes why stronger models gain less from compression and lose more from relay information loss, unifying prior empirical observations.
  • Controlled experimental design: three-way comparison (SAS / SAS-contextflow / MAS) across 5 benchmarks and 3 model scales (18 total comparisons) to isolate relay compression from decomposition effects.

Results

  • Low relay complexity (\(\delta \approx 0\)): ALFWorld, WideSearch, TravelPlanner-CS โ€” MAS consistently outperforms SAS-contextflow across all three model scales. Example gains at Qwen2.5-7B: +0.157 (ALFWorld SR), +0.194 (WideSearch Item F1), +0.183 (TravelPlanner-CS).
  • Higher relay complexity (\(\delta > 0\) or \(\delta \gg 0\)): WebShop, WorkBench, TravelPlanner-HC โ€” MAS gains shrink or reverse with model capability:
  • WorkBench (\(\delta \gg 0\)): MAS gain goes โˆ’0.005 โ†’ โˆ’0.086 โ†’ โˆ’0.014 across Qwen2.5-7B โ†’ GPT-4o-mini โ†’ Qwen3.5-27B.
  • TravelPlanner-HC: gain collapses from +0.017 (weak) to โˆ’0.233 (strong).
  • WebShop: MAS gain +0.080 at Qwen2.5-7B reverses to โˆ’0.003 at Qwen3.5-27B.
  • Pattern is consistent: MAS helps weaker models more when relays are near-sufficient; stronger models are hurt by relay information loss in high-\(\delta\) tasks.

Limitations

  • \(\beta\) and \(\delta\) are not directly measurable; they serve as qualitative/interpretive parameters rather than computable quantities to optimize against.
  • The framework assumes a sequential (chain) MAS topology; more complex communication graphs (e.g., hierarchical or peer-to-peer MAS) are not addressed.
  • Local sufficiency does not imply global sufficiency across long sequential chains when there are hidden long-range dependencies between early and late worker targets.
  • Experiments use only three model scales (Qwen2.5-7B, GPT-4o-mini, Qwen3.5-27B-AWQ-4bit); the ordering is treated as a monotone capability axis, which may not hold across architectures.
  • The framework does not prescribe how to design relays that optimally balance the trade-off; it explains when MAS helps but not how to achieve it.

Relevance to Agentic AI / LLM Agents

This paper provides the first principled theoretical account of MAS-vs-SAS performance variation, replacing ad hoc empirical claims with a mechanistic IB framework โ€” directly relevant to anyone designing or evaluating agentic pipelines. The result that MAS gains are not about decomposition per se, but about whether bounded inter-agent communication compresses noise without losing task-relevant signal, has immediate design implications: relay design (prompts, summarization strategies) should be the primary optimization target, not the number of agents. The capability-dependent \(\beta\) formally explains the widely observed empirical trend that stronger models benefit less from multi-agent architectures, connecting to the broader debate over when to use MAS versus stronger single-model inference.