Skip to content

AgentSynth: Scalable Task Generation for Generalist Computer-Use Agents

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Dawn Song (Berkeley); scalable automatic task/trajectory synthesis for computer-use agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AgentSynth is a fully automated pipeline for generating large-scale computer-use task datasets by chaining simple LLM-executed subtasks into hard long-horizon benchmarks. It exploits information asymmetry: subtasks are easy to generate and execute forward, but the composed task is hard to solve from scratch. The resulting benchmark exposes a stark human–agent gap, with SOTA models dropping from 18% success at difficulty level 1 to 4% at level 6, versus 70% human performance on level 6.

Problem

Human-annotated datasets for generalist computer-use agents (OSWorld, TheAgentCompany, τ-bench) are expensive (\(8–\)425/task), labor-intensive, and unscalable. Existing synthetic pipelines either retroactively label trajectories (OS-Genesis, Learn-by-interact) or generate tasks without reliable trajectory collection, and they fail to produce diverse, long-horizon tasks across multiple software domains.

Method

AgentSynth operates in the OSWorld gym environment and coordinates six LLM-based agents:

  1. Task Proposer — given a random persona from PersonaHub and the initial Ubuntu desktop screenshot, generates a short, atomic initial task.
  2. Task Executor — a two-stage ReAct agent: GPT-4.1 as planner (natural language next action) + computer-use-preview as grounder (pixel-level pyautogui commands); max 10 steps per subtask.
  3. Task Verifier — WebJudge-style: extracts key requirements, filters key screenshots (downsampled to 960×480), outputs binary success + completion percentage (Cohen's κ = 0.74 with human annotators).
  4. Task Reviser — if execution is partial, rewrites the task description to match what was actually completed, allowing the trajectory to remain useful.
  5. Follow-up Task Proposer — given the full prior subtask history and the latest screenshot, proposes the next logical subtask; repeats \(n\) times.
  6. Task Summarizer — merges the first \(k \in \{1,\ldots,n\}\) subtasks into a single high-level composite task, producing difficulty levels 1 through \(n+1\) from one execution chain.

The core insight is information asymmetry: generating tasks incrementally in the forward direction is tractable for current LLMs, while reconstructing the full multi-step plan from a single composite prompt is not.

Key Contributions

  • Fully automated pipeline producing >6,000 diverse, realistic computer-use tasks at $0.60/trajectory — orders of magnitude cheaper than human annotation.
  • Principled, fine-grained difficulty control: composing \(k\) subtasks yields difficulty level \(k\), scaling average horizon from 5 steps (level 1) to 45 steps (level 6) and memory span from 2 to 18 steps.
  • A calibrated LLM verifier with 88% agreement with human judges, 12% false-positive rate on near-miss adversarial variants, and 94% correct acceptance on benign UI-change variants.
  • A benchmark showing SOTA multimodal LLMs (GPT-4.1, Gemini 2.5 Pro, Claude 3.7 Sonnet, o3-mini) struggle severely, revealing a large human–agent performance gap.
  • Demonstrated generalizability to web agent environments (InSTA) beyond the primary OSWorld desktop setting.

Results

  • Human performance on level 6: 70% success rate.
  • SOTA model performance:
  • All models decline steeply from level 1 to level 6.
  • Best model at level 1: ~18% success rate.
  • Best model at level 6: ~4% success rate.
  • Models evaluated: GPT-4.1, o3-mini, Gemini 2.5 Pro, Claude 3.7 Sonnet.
  • Human evaluation of pipeline quality (100 sampled tasks):
  • Feasibility/realism: 91%; Subtask coherence: 90%; Persona relevance: 94%; Verifier accuracy: 88%.
  • Cost: $0.60/trajectory vs. \(8.80–\)110 (OSWorld), \(34–\)425 (TheAgentCompany).
  • Multi-app compositionality: >60% of trajectories involve ≥2 apps; >40% involve ≥3 apps.
  • At level 6: avg. 3.3 applications used, 4.3 app switches per task.

Limitations

  • Task generation relies exclusively on GPT-4.1; different generator models may produce systematically different complexity or realism, and the sensitivity is unexplored.
  • Pipeline is built on OSWorld's simulated Ubuntu desktop; generalization to other OS environments (Windows, macOS) is not demonstrated in the main evaluation.
  • Each subtask is capped at 10 steps, which may cause the reviser to degrade task quality when tasks consistently exceed that budget.
  • LLM verifier shows mild accuracy decline at higher difficulty levels and a non-trivial 12% false-positive rate on near-miss failures.
  • Evaluation uses bare LLMs without agent scaffolding, representing a lower bound; performance of production-grade agents on this benchmark is not reported.
  • No downstream training experiments showing whether AgentSynth trajectories improve agent policy performance.

Relevance to Agentic AI / LLM Agents

AgentSynth directly addresses the data bottleneck for training and evaluating generalist computer-use agents — a core infrastructure problem in the field. The information-asymmetry principle (forward generation is easy; holistic solving is hard) is a transferable design pattern for bootstrapping hard benchmarks from cheap synthetic forward passes, relevant beyond computer-use to any sequential decision-making domain. The steep human–SOTA gap (70% vs. 4% at level 6) provides a concrete, calibrated measure of how far current LLM agents are from human-level long-horizon computer control. For researchers building or evaluating agentic systems, this work offers both a scalable data recipe and a challenging new benchmark that can discriminate between agents at high difficulty levels where existing benchmarks saturate.