Skip to content

FutureWeaver: Planning Test-Time Compute for Multi-Agent Systems with Modularized Collaboration

🕒 Published (v1): 2025-12-12 01:43 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

FutureWeaver is a framework that optimizes test-time compute allocation for multi-agent systems under fixed monetary budgets. It introduces automatically-induced collaboration modules (reusable multi-agent workflow primitives) and a dual-level planner that combines short-horizon action scoring with speculative long-horizon budget feasibility lookahead. On GAIA, BrowseComp-Plus, and BigCodeBench, it consistently outperforms ReAct and fixed-workflow baselines across all budget tiers.

Problem

Standard test-time scaling (Best-of-N, iterative verification) applies within a single model invocation and does not extend to multi-agent settings, where compute must be distributed across heterogeneous agent interactions. Existing orchestrator–worker systems execute agents sequentially with no principled mechanism for budget-aware coordination or for exploiting complementary agent capabilities. This leaves available budget underutilized and agent synergies unexploited.

Method

FutureWeaver extends the orchestrator–worker paradigm with three components:

  1. Collaboration Modules: Formalized as m = (S, κ) where S ⊆ A is a subset of agents and κ is a coordination strategy (e.g., ensemble reasoning, pipeline chaining). These are callable like functions, expanding the orchestrator's action space from A to A' = A ∪ M.

  2. Dual-Level Planning: At each step, K candidate actions are sampled. A short-horizon gain g(α) scores each candidate by the geometric mean of self-consistency (modal agreement across K samples) and a performance prior (trajectory-level success rate retrieved via semantic similarity from an experience buffer). A long-horizon feasibility score h(α) expands each candidate into symbolic module-level speculative rollouts, filters those exceeding remaining budget B_t, and normalizes the count of feasible continuations. The orchestrator selects argmax f(α) = g(α) + h(α), and the feasible speculative set is carried forward to bias the next step's proposal distribution (A*-like coupling).

  3. Self-Play Reflection: An offline iterative process that runs trajectories on a validation subset, collecting per-agent/module cost and success statistics, and prompts an LLM to abstract recurring successful interaction sequences into new collaboration modules. Five rounds of self-play are run per benchmark.

Key Contributions

  • Formal problem definition: budget-constrained test-time compute optimization for multi-agent orchestration.
  • Collaboration modules as a reusable, composable abstraction for multi-agent coordination, automatically induced via self-play reflection without manual engineering.
  • Dual-level planner integrating short-horizon self-consistency/performance-prior scoring with lightweight symbolic long-horizon budget feasibility lookahead.
  • Demonstration that baselines consistently underutilize available budget, while FutureWeaver achieves substantially higher cost utilization, converting headroom into accuracy.

Results

  • GAIA (Claude models): FutureWeaver reaches 48.15% Acc@$0.5 vs. 43.20% for Budget-Aware collaboration baseline, 38.27% for ReAct+IterVer, and 12.96% for AFlow.
  • BrowseComp-Plus (Claude): 29.00% Acc@$0.5 vs. 28.07% Budget-Aware, 24.80% ReAct+BoN, 5.42% AFlow.
  • BigCodeBench (Claude): 48.2% Acc@$0.1 vs. 47.2% Budget-Aware, 46.2% ReAct+IterVer, 46.1% AFlow.
  • Qwen3-32B (GAIA): 21.33% Acc@$0.3 vs. 14.00% Budget-Aware, 16.67% ReAct+IterVer.
  • Ablation (GAIA Claude, Acc@$0.5): Short-term only 44.44%, Long-term only 46.91%, Dual-level 48.15%.
  • Budget utilization plots confirm baselines plateau well below the budget ceiling while FutureWeaver tracks closer to the constraint.

Limitations

  • Self-play reflection requires a held-out validation set per benchmark (30 questions used), introducing a data-setup overhead before deployment.
  • Cost estimates for collaboration modules are averaged over observed subtasks, with acknowledged approximation error (gap analyzed in an appendix but not reported in the main text).
  • BigCodeBench gains are modest (≤2 pp) compared to GAIA/BrowseComp-Plus; the paper notes coding tasks favor end-to-end workflows.
  • Self-play produces duplicate modules in later rounds (BrowseComp-Plus and BigCodeBench yield <5 unique modules from 5 rounds), suggesting limited module diversity for some domains.
  • Evaluation is restricted to three benchmarks; generalization to other multi-agent task types is not demonstrated.

Relevance to Harnesses / Meta-Harnesses

FutureWeaver is a meta-harness in the strict sense: its primary object is not task execution but the orchestration policy—deciding which agent or workflow module to call, when, and in what order, subject to a hard resource budget. The collaboration module abstraction (reusable, callable multi-agent workflow primitives induced from experience) is directly analogous to tool libraries in harness design, but auto-generated rather than hand-authored. The dual-level planner operationalizes a principle central to meta-harness design: local action selection must be coupled to global resource accounting. Self-play reflection as an offline harness-calibration step—gathering cost profiles and distilling workflow patterns before live deployment—offers a concrete mechanism that harness designers could adopt to auto-tune module repertoires and cost models.