SCALECUA: Scaling Computer Use Agents with Verifiable Task Synthesis and Efficient Online RL¶

🕒 Published (v1): 2026-07-13 07:32 UTC · Source: Arxiv · link

Why this paper was selected

Xiao Liu (AgentBench) group; RLVR + verifiable task synthesis for computer-use agents at scale

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

SCALECUA is a unified framework for scaling computer use agents (CUAs) via two complementary advances: VERIGEN, a multi-agent pipeline that synthesizes verifiable GUI tasks at scale from live OS containers, and efficient online RL with Frontier Sampling and Visual Context Segmentation. Trained on Qwen3.5-9B, it achieves 68.7% on OSWorld and 54.0% on ScienceBoard—new state-of-the-art among open-source CUAs, surpassing models up to 4× larger.

Problem¶

Online RL with verifiable rewards (RLVR) for CUAs is bottlenecked by two gaps: (1) data scarcity—GUI tasks lack built-in verifiers (unlike math/code), and randomly synthesized tasks are poorly matched to the model's current capability, wasting rollouts under GRPO-style training; (2) training inefficiency—trajectory-level training accumulates all screenshots (slowing the rollout engine), while step-wise decomposition inflates training items by $O(T)$ per trajectory (slowing the training engine).

Method¶

VERIGEN (verifiable task synthesis): Three LLM agents (proposer, judger, checker) iteratively synthesize tasks and executable judge functions against live Docker containers. A shared "docker interaction probe" exposes environment I/O to 100+ parallel agent workers without direct container entry. Trajectory-guided synthesis further decomposes failed tasks into easier subtasks and composes succeeded tasks into harder multi-objective ones. Output: 24K+ verifiable tasks, ~3K RL-ready tasks.

Frontier Sampling: Maintains a per-task EMA success rate $\hat{p}_i$ updated as: $$\hat{p}_i \leftarrow (1 - \alpha)\hat{p}_i + \alpha \cdot \frac{1}{n}\sum_{j=1}^{n} \mathbf{1}[\text{rollout}_j \text{ succeeds}]$$ Sampling weight is assigned via a Gaussian kernel centered at $\mu=0.5$: $$w_i = \exp\!\left(-\frac{(\hat{p}_i - \mu)^2}{2\sigma^2}\right)$$ with $\sigma=0.25$, concentrating rollouts on tasks at the model's current learning frontier. A fraction $\gamma=0.2$ per batch is reserved for uniform exploration.

Visual Context Segmentation: A sliding window over the visual token stream retains the $K$ most recent screenshots while preserving full text history. When the window overflows by $\Delta$ screenshots, the current token-ID stream is emitted as a training item, oldest images are dropped, and the stream is re-tokenized. This yields $O(nT/K)$ training items—between trajectory-level $O(n)$ and step-wise $O(nT)$—with $K=5$–$8$ found optimal across compute scales.

Key Contributions¶

VERIGEN: end-to-end pipeline for scalable verifiable GUI task synthesis via iterative multi-agent + live Docker interaction; achieves 94.5% judge executable rate.
Frontier Sampling: lightweight EMA-based task sampler that allocates rollouts to the model's current capability frontier; ablation shows +5.0% OSWorld over uniform sampling baseline.
Visual Context Segmentation: sliding-window multi-turn trajectory scheme achieving 2.83× end-to-end training speedup over step-wise decomposition without sacrificing cross-step reasoning.
State-of-the-art open-source CUA results: 68.7% OSWorld, 54.0% ScienceBoard with a 9B model.

Results¶

OSWorld (overall): SCALECUA-Qwen3.5-9B 68.7% vs. Kimi K2.5 63.3%, EvoCUA-32B 56.7%, ComputerRL-9B 48.0%, Claude Sonnet 4.5 62.9% (proprietary).
OSWorld (base → trained): Qwen3.5-9B improves from 41.8% → 68.7% (+26.9 pp); Qwen3-VL-8B from 33.9% → 67.7%.
ScienceBoard (overall): SCALECUA-Qwen3.5-9B 54.0% vs. Claude Opus 4.6 52.7% (all prior open-source ≤ 20.4%).
ScienceBoard subdomains: TeXstudio 86.7%; Lean theorem proving 19.0% (vs. ≤15.4% for all proprietary baselines).
Ablation (OSWorld): Full 68.7% vs. w/o VERIGEN 43.9%, w/o Frontier Sampling 63.7%, w/o Visual Context Segmentation 62.2%.
VERIGEN judge executable rate: Full pipeline 94.5% vs. w/o LLM Judge Agent 62.3%, w/o Fix Agent 78.1%, w/o Rule Validator 86.2%.
Training speedup: Visual Context Segmentation achieves 2.83× over step-wise decomposition.
Scaling generated tasks from 0K → 20K raises OSWorld performance monotonically (Figure 1 right).

Limitations¶

Verifiable task synthesis currently targets full desktop OS environments (OSWorld, ScienceBoard); generalization to mobile or browser-only environments is not evaluated.
The parallel Docker probe infrastructure (100+ concurrent containers) requires significant compute and cloud resources, limiting reproducibility at small scale.
Frontier Sampling hyperparameters ($\mu$, $\sigma$, $\gamma$, $\alpha$) are fixed across all experiments; sensitivity to task-distribution shift is not analyzed.
Visual Context Segmentation discards older screenshots, potentially losing visual state relevant to long-horizon tasks beyond the window; mitigated but not eliminated by textual history retention.
ScienceBoard GIS (GrassGIS) scores remain low (≤19.0%), indicating persistent difficulty for domain-specific GUI interactions.

Relevance to Agentic AI / LLM Agents¶

SCALECUA directly addresses the core bottleneck in training autonomous GUI agents: the lack of scalable verifiable reward signal in open-ended desktop environments. The VERIGEN pipeline generalizes the RLVR recipe—proven in math and code domains—to live OS containers, establishing a replicable path for growing CUA training corpora without human annotation. Frontier Sampling is a practical curriculum mechanism applicable to any multi-task online RL setting where task difficulty is heterogeneous, and Visual Context Segmentation solves a general problem in long-horizon multimodal RL: how to balance visual context fidelity against rollout and training cost. For researchers tracking LLM agents, this work demonstrates that a 9B open-source model trained with the right data and RL infrastructure can match or surpass proprietary systems on realistic multi-step desktop tasks.