Skip to content

Scaling Synthetic Task Generation for Agents via Exploration

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; Meta/CMU; scaling synthetic task generation for MLLM agent post-training

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AutoPlay is a two-stage pipeline that trains an MLLM explorer agent to exhaustively interact with UI environments, then uses those exploration trajectories as grounded context for a task-generator MLLM to synthesize diverse, feasible, and verifiable tasks at scale. It produces 20k Android and 10k Ubuntu tasks without human annotation, enabling SFT and RL post-training that lifts UI agent success rates by up to 20% (mobile) and 10.9% (desktop).

Problem

Post-training MLLM-based UI agents (computer-use, mobile-use) requires large, diverse, and verifiable task datasets, but existing methods either rely on costly human annotation or prompt MLLMs with minimal environment context—yielding generic tasks with poor coverage, unknown feasibility, and no grounding in actual environment state.

Method

AutoPlay operates as a two-stage pipeline formalized over POMDPs \((S, O, A, P, R, \rho_0)\):

Stage 1 – Environment Exploration: A goal-agnostic MLLM explorer agent, equipped with an episodic memory of summarized prior trajectories \(\{m_1, \ldots, m_{i-1}\}\), interacts with the environment for \(K\) steps per turn and \(M\) turns per app, producing exploration trajectories \(E = \{\tau_1, \ldots, \tau_M\}\). Summaries are generated by a separate summarizer MLLM to fit within context windows.

Stage 2 – Exploration-Conditioned Task Generation: A task-generator MLLM receives each trajectory \(\tau \in E\) as environment context (exposing feasible interactions and current state) together with one of \(N\) task guideline prompts \(P = \{p_1, \ldots, p_N\}\) (e.g., Feature-Use CRUD, Information Retrieval, Feature Composition, Subtask Repetition). It samples \(K\) tasks \((g_1, \ldots, g_K)\) per \((τ, p)\) pair, each paired with the trajectory's initial state \(s_0\) to form task dataset \(D\).

Task demonstrations are collected by a GPT-4o planner + grounding model (UI-TARS-1.5 7B for mobile, GTA1-7B for desktop) executor agent and verified by a GPT-4o MLLM verifier using chain-of-thought reasoning. SFT is applied on verified trajectories; RL uses the MLLM verifier as a binary reward signal (\(r \in \{0,1\}\)) with GRPO on 32 H100 GPUs. Base models are Qwen2.5-VL-Instruct (3B–72B).

Key Contributions

  • AutoPlay pipeline that decouples exploration from task generation, enabling a single trajectory to spawn multiple tasks via different guideline prompts.
  • Memory-augmented MLLM explorer that maximizes novel state coverage across \(M\) turns via episodic summarization.
  • Fully annotation-free SFT+RL pipeline: MLLM executor generates demonstrations; MLLM verifier provides rewards—no privileged environment access required.
  • 20k Android tasks (20 apps) and 10k Ubuntu tasks (13 apps) with ~8k and ~3.5k verified trajectories respectively.
  • Demonstrated that exploration-grounded tasks outperform prior iterative and description-based synthesis methods on task diversity, coverage, and downstream agent performance.

Results

  • AndroidWorld (mobile-use):
  • AutoPlay-7B: Pass@1 = 40.1 (+20.6 vs. Qwen2.5-VL-7B base at 19.5), Pass@5 = 52.2
  • AutoPlay-72B: Pass@1 = 47.9 (+12.9 vs. base 35.0), Pass@5 = 68.2
  • AutoPlay-3B RL: Pass@1 = 39.9 (+5.7 over AutoPlay-3B SFT at 34.2), Pass@5 = 53.4
  • OSWorld (computer-use):
  • AutoPlay-7B: Pass@1 = 11.4 (+7.7 vs. Qwen2.5-VL-7B base at 3.7), Pass@5 = 12.1
  • AutoPlay-72B: Pass@1 = 14.5 (+10.1 vs. base 4.4), Pass@5 = 16.0
  • AutoPlay-72B on AndroidWorld (47.9) approaches GPT-4o + UI-TARS (43.1) and UI-TARS-1.5 72B (37.7), approaching much larger/proprietary systems.
  • RL on top of SFT yields an additional 5.7% gain (3B model on AndroidWorld).

Limitations

  • Evaluation is restricted to two UI platforms (Android, Ubuntu); generalization to other agentic domains (web, robotics) is not demonstrated.
  • Task verifier is an MLLM (GPT-4o) without privileged environment access; verifier errors propagate into both SFT data quality and RL reward signal.
  • Exploration relies on MLLM explorer quality; rare or deeply nested app states may remain undiscovered within \(M\) turns.
  • RL training is restricted to the ~8k tasks the executor can solve at least once, limiting coverage of harder tasks during RL.
  • Heuristic actions used by the executor for complex UI elements (e.g., spreadsheets) are withheld from the trained policy, creating a train/test gap.

Relevance to Agentic AI / LLM Agents

AutoPlay directly addresses the data bottleneck for post-training agentic MLLMs—a core open problem for any agent that must operate in closed, interactive environments where web-scale data is unavailable. The separation of environment exploration from task synthesis, combined with MLLM-based verification as a reward model, offers a general recipe for annotation-free agent training applicable beyond UI domains. The RL integration (GRPO with a verifier reward) demonstrates that synthetically generated tasks can serve as a scalable curriculum for reinforcement learning, complementing the SFT paradigm. For practitioners building GUI or computer-use agents, AutoPlay provides a concrete, reproducible pipeline that outperforms prior iterative task-generation baselines (PAE, OSUniverse, Trabucco et al.) on diversity and downstream performance.