Skip to content

OWL: Optimized Workforce Learning for General Multi-Agent Assistance in Real-World Task Automation

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

WORKFORCE is a hierarchical multi-agent framework that decouples domain-agnostic planning from domain-specific execution, enabling plug-and-play cross-domain transfer without full system redesign. Paired with Optimized Workforce Learning (OWL)—a two-stage SFT+DPO training scheme applied exclusively to the planner—the system achieves 69.70% on GAIA (open-source SOTA, surpassing OpenAI Deep Research), and OWL boosts Qwen2.5-32B-Instruct by +16.37 pp.

Problem

Existing LLM multi-agent systems are domain-specific: deploying them in new domains requires complete architectural redesign (e.g., MetaGPT's hard-coded SOPs) and retraining every agent component (e.g., MALT's generator-verifier-refiner pipeline). This rigidity prevents scalable generalization across diverse real-world task domains.

Method

WORKFORCE introduces a three-tier hierarchy: (1) a domain-agnostic Planner that decomposes tasks into subtasks based on a worker capability registry; (2) a Coordinator that assigns subtasks, manages dependencies, and integrates results via a shared task channel; (3) domain-specific Worker Nodes (Web Agent, Document Agent, Reasoning/Coding Agent) with isolated execution contexts that post only final results back to the channel. A replanning mechanism allows the Planner to regenerate subtasks upon worker-reported failures, enabling test-time scaling.

For training, OWL targets only the Planner with: (i) SFT on 1,599 filtered expert trajectories synthesized via GPT-4o-mini across HotpotQA, WikiTableQuestions, math, and Infinity-MM; (ii) DPO on 1,009 preference pairs constructed from n=4 rollouts per question, scored by accuracy or LLM-as-judge. Workers remain frozen; only the planner is updated.

Key Contributions

  • WORKFORCE architecture: modular Planner–Coordinator–Worker hierarchy enabling plug-and-play domain transfer at inference time by swapping Worker Nodes.
  • OWL training paradigm: SFT+DPO applied solely to the domain-agnostic planner, avoiding full system retraining and reducing computational cost.
  • Open-source SOTA on GAIA: 69.70% overall accuracy, first open-source system to exceed OpenAI Deep Research (67.36%).
  • OWL-trained Qwen2.5-32B-Instruct: 52.73% (+16.37 pp over base), outperforming GPT-4o-mini (47.27%) and Qwen2.5-72B-Instruct (49.09%).
  • Full open-source release of code, models, and training data.

Results

  • WORKFORCE (Claude-3.7-Sonnet): 69.70% on GAIA validation; surpasses OpenAI Deep Research (67.36%) by +2.34 pp and all prior open-source systems.
  • WORKFORCE (GPT-4o): 60.61%; +23.03 pp over Single Agent baseline and +6.06 pp over Role Playing multi-agent baseline (same model and tools).
  • OWL (Qwen2.5-32B-Instruct): 52.73% after SFT+DPO vs. 36.36% base (+16.37 pp); surpasses GPT-4o-mini (47.27%) and Qwen2.5-72B-Instruct (49.09%); matches GPT-4o on Level 3 tasks (26.92% vs. 26.92%).
  • SFT alone improves Level 1/2 but causes -3.85 pp regression on Level 3; adding DPO recovers and exceeds base across all levels.
  • Trajectory filtering: filtered training consistently outperforms unfiltered (52.73% vs. 41.21%), confirming data quality dominates quantity.
  • Planner-only training (45.45%) substantially outperforms worker-only training (31.51%); joint training (46.68%) adds minimal gain at much higher cost.
  • Replanning: performance improves monotonically with replanning count (0→1→2 iterations) for both GPT-4o and Claude-3.7-Sonnet workforces.
  • Robustness: Role Playing degrades from 62.3% to 34.6% as required capabilities increase from 1 to ≥3; WORKFORCE maintains stable performance across all complexity levels.

Limitations

  • GAIA validation set contamination risk required blocking specific websites; generalization to fully held-out benchmarks is not demonstrated.
  • OWL training uses pass@3 for WORKFORCE inference but pass@1 for the trained model evaluation, making direct comparison slightly asymmetric.
  • SFT alone regresses on Level 3 tasks (-3.85 pp), indicating instability without the DPO stage.
  • Training data is modest in scale (1,599 SFT / 1,009 DPO pairs), raising questions about further scaling behavior.
  • Error analysis shows ~50% of failures stem from foundation model or tool limitations outside the framework's control, capping the ceiling achievable via planner optimization alone.
  • Only three worker types instantiated; extension to broader domains (e.g., specialized scientific tools) is described but not empirically validated.

Relevance to Agentic AI / LLM Agents

WORKFORCE directly addresses the central unsolved problem of cross-domain generalization in multi-agent systems by isolating the planning bottleneck and showing that RL-trained domain-agnostic planners transfer better than full-system fine-tuning. The OWL result—a 32B open model matching GPT-4o on hard tasks via targeted planner-only DPO—is directly relevant to the line of work on post-training agentic LLMs (WebRL, Search-R1, ReTool), but distinguishes itself by training within a multi-agent scaffold rather than on single-agent trajectories. The replanning-as-test-time-scaling finding also connects to broader interest in inference-time compute scaling for agents. For practitioners building general-purpose LLM agent systems, this paper offers a concrete architectural recipe and training strategy validated on a rigorous real-world benchmark.