Skip to content

OWL: Optimized Workforce Learning for General Multi-Agent Assistance in Real-World Task Automation

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

WORKFORCE is a hierarchical multi-agent framework that decouples domain-agnostic planning from domain-specific execution via a Planner → Coordinator → Worker Node stack, achieving 69.70% on the GAIA benchmark (open-source SOTA, surpassing OpenAI Deep Research by 2.34%). OWL (Optimized Workforce Learning) then trains only the Planner with SFT + DPO on a cross-domain curriculum, lifting a Qwen2.5-32B-Instruct planner from 36.36% to 52.73% (+16.37%) without retraining workers.

Problem

Existing LLM multi-agent systems are domain-specific: inference deployment requires full architectural redesign (e.g., MetaGPT's SOPs), and training must retrain every agent component (e.g., MALT's generator-verifier-refiner pipeline). This makes cross-domain transfer prohibitively expensive and prevents building a general-purpose, reusable multi-agent substrate.

Method

WORKFORCE (inference): Three-layer hierarchy communicating via a shared Task Channel: 1. Planner — decomposes the user task into subtasks based on registered worker capabilities. 2. Coordinator — dispatches subtasks to workers, tracks dependencies, integrates results. 3. Worker Nodes — specialized agents with domain-specific toolkits (Web Agent, Document Agent, Reasoning/Coding Agent); workers report only final results to the channel, keeping per-worker context clean.

A replanning mechanism triggers when a worker self-reports subtask failure: the planner receives the failure signal and synthesizes new subtasks, enabling test-time scaling.

OWL (training): Two-phase optimization applied exclusively to the Planner: - Phase 1 — SFT: GPT-4o-mini rollouts on four curriculum datasets (HotpotQA, WikiTableQuestions, Math, Infinity-MM) produce expert trajectories. Quality filtering (accuracy, cosine similarity ≥ 0.7, LLM-as-judge) yields 1,599 trajectories averaging 3.41 subtasks each. - Phase 2 — DPO: The SFT model rolls out \(n=4\) trajectories per question; correct/high-similarity outcomes are "chosen", others "rejected". 1,009 preference pairs are used for Direct Preference Optimization.

Training uses Qwen2.5-32B-Instruct on 8× H100 GPUs, 2 epochs, max 32,768-token context, LR \(= 10^{-5}\), bfloat16, effective batch size 12.

Key Contributions

  • Modular hierarchical MAS architecture (WORKFORCE) enabling plug-and-play worker substitution without touching the planner/coordinator.
  • OWL training paradigm: trains only the domain-agnostic planner (not workers), drastically reducing retraining overhead for new domains.
  • Open-source SOTA on GAIA: 69.70% (Claude-3.7-Sonnet backend), first open-source system to exceed OpenAI Deep Research.
  • Efficient planner post-training: Qwen2.5-32B achieves +16.37% and matches GPT-4o on Level 3 tasks (26.92% vs. 26.92%) with a small filtered dataset.
  • Full open-source release of code, models, and data.

Results

  • WORKFORCE (Claude-3.7-Sonnet): 69.70% on GAIA validation; L1: 84.91%, L2: 68.60%, L3: 42.31%.
  • WORKFORCE (GPT-4o, pass@3): 60.61%, vs. Single Agent 37.58% (+23.03%) and Role Playing 52.33% (+6.06%) — same toolkits.
  • Surpasses OpenAI Deep Research (67.36%) by +2.34% and Langfun Agent v2.1 (69.70%) ties at Level 1 with +1.89% on Level 1 specifically.
  • OWL Qwen2.5-32B-Instruct: base 36.36% → SFT 41.21% → SFT+DPO 52.73% (+16.37%).
  • SFT alone regresses on Level 3 (−3.85%); DPO recovers and adds +7.69% on Level 3 relative to base.
  • OWL-trained 32B outperforms GPT-4o-mini (47.27%) and Qwen2.5-72B-Instruct (49.09%).
  • Trajectory filtering vs. no filtering: 52.73% vs. 50% (filtered consistently better, confirming quality > quantity).
  • Replanning up to 2 iterations improves WORKFORCE score from 0.43 → 0.60; ≥3 iterations plateau.

Limitations

  • SFT alone causes regression on hardest tasks (Level 3: −3.85%), requiring DPO to recover generalization.
  • WORKFORCE (GPT-4o) uses pass@3 vs. pass@1 for other baselines, making direct comparison potentially optimistic.
  • OWL-trained 32B still trails GPT-4o planner (52.73% vs. 60.61%) overall.
  • Failure analysis shows persistent failure modes: context overflow (19.2%), incorrect plan (19.2%), limited tool capability (11.5%), multimodal toolkit failures — indicating worker-side bottlenecks that OWL (planner-only training) cannot address.
  • Curriculum datasets are limited to four domains; breadth of generalization beyond these is unvalidated.
  • Evaluation uses GAIA validation set only; test-set performance is unreported.

Relevance to Harnesses / Meta-Harnesses

WORKFORCE is architecturally a meta-harness: a runtime that orchestrates heterogeneous specialized sub-agents through a shared task channel, with a centralized planning/coordination layer sitting above the workers. Its explicit separation of the orchestration core (Planner + Coordinator) from the execution periphery (Workers) is the defining structural pattern of meta-harnesses, and the replanning loop is a canonical harness-level error-recovery mechanism. OWL advances the harness line of work by demonstrating that only the orchestration layer needs training for cross-domain transfer — a "stable core, variable periphery" principle with direct implications for designing trainable meta-harnesses where tools/workers remain modular plug-ins. The curriculum-based SFT + DPO pipeline for planner optimization is a concrete, reproducible recipe for training the control plane of a general-purpose harness.