Skip to content

MMoA: An AI-Agent framework with recurrence for Memoried Mixure-of-Agent

🕒 Published (v1): 2026-05-18 23:47 UTC · Source: Arxiv · link

Why this paper was selected

MMoA: recurrent memoried mixture-of-agent with dynamic context-aware routing

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MMoA augments the Mixture-of-Agents (MoA) framework with an LSTM-based recurrent router that dynamically selects and weights agent contributions using both current inputs and historical routing state across aggregation layers. This replaces the static aggregation of standard MoA, reducing inference cost while preserving most accuracy. On AlpacaEval 2.0, MMoA achieves 58.0% win rate vs. MoA's 59.8% with up to 4.6% runtime reduction.

Problem

Standard MoA uses a static router that aggregates agent outputs independently at each layer, ignoring temporal and contextual dependencies across successive aggregation layers. This prevents the system from learning which agents are most valuable given the evolving context, and forces all agents to be invoked uniformly regardless of their marginal utility at each depth.

Method

MMoA introduces a recurrent gating module at each MoA aggregation layer. Given \(n\) agent outputs \(\{A_1(x), \ldots, A_n(x)\}\), they are fused via concatenation + linear projection into \(z_t = f(A_1(x), \ldots, A_n(x))\). An LSTM then updates a hidden state:

\[h_t = \text{LSTM}(z_t,\, h_{t-1})\]

A gating vector is computed as \(g = \text{softmax}(W_g h_t + b_g) \in \mathbb{R}^n\), and the aggregated output is the weighted sum \(y = \sum_{i=1}^n g_i \cdot A_i(x)\). This hidden state propagates across all \(L\) layers, reducing effective time complexity from \(O(nL)\) to \(O(n+L)\). The router is trained with an expected task loss \(\sum_i p_i L_i\) plus entropy regularization \(\lambda H(p)\) and optional load-balancing term \(\gamma L_\text{load}\).

Key Contributions

  • First introduction of a trainable router for agent selection within the MoA framework.
  • LSTM-based recurrent gating module that conditions routing on historical hidden state across aggregation layers.
  • A router loss combining expected task loss, entropy regularization, and optional load-balancing to train the selection distribution end-to-end.
  • Demonstrated accuracy-efficiency trade-off: minimal accuracy drop with measurable runtime reduction by activating fewer agents per layer.

Results

  • AlpacaEval 2.0: MMoA LC win rate 61.5% vs. MoA 65.1% (−3.6 pts); overall win rate 58.0% vs. 59.8% (−1.8 pts).
  • MT-Bench: MMoA avg. score 9.20 vs. MoA 9.25 (−0.05); per-turn drops under 0.05.
  • Inference time (multiple-proposer vs. single-proposer baseline):
  • \(n=6\): 56.7% vs. 61.3% of baseline (−4.6% absolute)
  • \(n=3\): 56.1% vs. 58.0% (−1.9%)
  • \(n=2\): 54.5% vs. 58.8% (−4.3%)
  • \(n=1\): 47.8% vs. 47.8% (no overhead)

Limitations

  • Evaluated only on instruction-following benchmarks; generalization to long-context reasoning, tool use, multilingual tasks, or safety-critical settings is untested.
  • Router is LSTM-only; Transformer-based, RL-based, or token-level routing alternatives are unexplored.
  • Additional hyperparameters (\(\lambda\), \(\gamma\), aggregation depth) and sensitivity to training data and agent pool composition are not systematically analyzed.
  • No investigation of qualitative failure modes, calibration, or robustness beyond win-rate metrics.

Relevance to Harnesses / Meta-Harnesses

MMoA is directly relevant as a learned meta-orchestration layer that decides at runtime which sub-agents to weight heavily across a multi-step pipeline — the core function of a meta-harness. The LSTM router acts as a stateful controller that observes the output of each agent layer and adaptively modulates the next layer's composition, analogous to how a meta-harness might route tasks between specialized sub-agents based on accumulated context. The explicit router loss formulation (with load-balancing) addresses a practical concern in harness design: preventing over-reliance on a single agent. This work provides a differentiable, trainable alternative to the rule-based or static routing logic found in most current meta-harnesses.