Skip to content

Stop Wasting Your Tokens: Towards Efficient Runtime Multi-Agent Systems

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR; targets token waste and misinformation in runtime multi-agent systems

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SUPERVISOR AGENT is a lightweight, non-intrusive meta-agent that monitors Multi-Agent System (MAS) interactions at runtime, intervening only when an LLM-free adaptive filter detects errors, inefficiency loops, or excessively long observations. On GAIA, it cuts Smolagent's token consumption by 29.68% on average at pass@1 with no loss in success rate. The framework is model- and MAS-agnostic, confirmed across GPT-4.1, Gemini-2.5-pro, Qwen3, and multiple MAS frameworks.

Problem

Modern MAS suffer from two compounding runtime failure modes: (1) error propagation, where a single hallucinated output poisons all downstream agents via shared memory or agent-agent communication; and (2) economic inefficiency, from verbose tool outputs flooding context windows and agents entering repetitive action loops. Existing work addresses robustness post-hoc (failure attribution after the fact) or via design-time architecture pruning—neither intervenes dynamically during execution.

Method

The framework introduces a Supervised Multi-Agent System (SMAS) with three components:

  1. LLM-free adaptive filter — heuristically monitors the three high-risk interaction points (agent-agent, agent-tool, agent-memory) and triggers supervision only on: explicit errors, detected repetitive/inefficient behavior patterns, or observations exceeding a length threshold. This avoids the overhead of inspecting every interaction.

  2. Memory-augmented context window \(W = (N, Q_g, Q_l, T_l, S)\) (and extended \(W_{ext} = W \cup \{T_g\}\) for system-wide diagnosis) — gives the supervisor a holistic view beyond any individual agent's local trace.

  3. Multi-level action space \(A(c)\) conditioned on intervention context \(c \in \{c_{error}, c_{inefficient}, c_{excessive}\}\):

  4. approve — permit productive-but-repetitive behavior to continue
  5. provide_guidance — append directive hint to steer agent reasoning
  6. correct_observation — replace raw observation with purified version
  7. run_verification — invoke a verification sub-agent for external fact-checking

SUPERVISOR AGENT wraps an existing MAS without modifying any base agent's architecture.

Key Contributions

  • Formal definition of SMAS: a MAS augmented with a meta-level control agent that monitors interactions in real time without architectural intrusion.
  • LLM-free adaptive filter enabling selective, low-overhead supervision at critical junctures.
  • Three-strategy intervention system (error correction, inefficiency guidance, observation purification) with ablation demonstrating complementary roles.
  • Demonstrated 29.68% average token reduction on GAIA (pass@1) with maintained accuracy; Pareto improvements across math reasoning (AIME +6.67% accuracy, −18.92% tokens), code generation (HumanEval −23.74% tokens), and QA benchmarks.
  • MAS-agnostic validation on Smolagent, AWorld, and OAgents; model-agnostic validation across GPT-4.1, Gemini-2.5-pro, and Qwen3-235B.

Results

  • GAIA (pass@1): 29.68% token reduction over Smolagent baseline; accuracy held at 50.91%; Level 2 token savings 34.64%, Level 3 29.23%.
  • GAIA (pass@3): 26.45% token reduction with accuracy improving from 61.82% → 63.03%.
  • GAIA Level 2 (pass@1): token cost variance reduced by 63% (from violin plot analysis).
  • AIME 2024: solve rate 30.00% → 36.67% (+6.67%), tokens 59.14K → 47.95K (−18.92%).
  • HumanEval: accuracy 92.07% → 92.68%; tokens −23.74% (40.91K → 31.19K).
  • GSM-Hard: accuracy 74.33% → 75.50%; tokens −8.92%.
  • DROP (F1): slight drop 81.08 → 79.80 (attributed to over-compression during purification).
  • Ablation: Purification drives the majority of token savings (removing it cuts savings from 29.68% to 15.96%); removing Correction or Guidance causes the largest accuracy drops.

Limitations

  • Observation purification can over-compress long contexts, causing minor accuracy or F1 drops on certain benchmarks (DROP F1 drops ~1.3 points).
  • The adaptive filter uses heuristic rules for three predefined scenarios; failure modes outside these categories (novel error types, unusual agent topologies) may not trigger supervision.
  • Overhead from SUPERVISOR AGENT's own LLM calls is not fully quantified against the savings in cases where supervision is invoked frequently on simpler tasks.
  • Evaluation on AWorld and OAgents uses only the top-10 most token-intensive GAIA tasks per level, limiting the generalizability evidence for non-Smolagent frameworks.
  • No evaluation on multi-modal or embodied MAS settings; all benchmarks are text-only.

Relevance to Harnesses / Meta-Harnesses

SUPERVISOR AGENT is a direct instantiation of the meta-harness pattern: a control layer that wraps and monitors an existing agent system without modifying its internals, using a filter-trigger-intervene loop rather than static pipeline configuration. The distinction between a heuristic-gated "when to supervise" phase and an LLM-driven "how to supervise" phase mirrors the architecture of harnesses that separate orchestration logic from model inference. The paper's formalism (SMAS, context window tuple, action space per intervention context) provides concrete vocabulary and design primitives for anyone building adaptive meta-harnesses, and the ablation study quantifies the cost-vs-robustness tradeoff of each harness module independently.