Skip to content

OpenCUA: Open Foundations for Computer-Use Agents

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

OpenCUA is a fully open-source framework for building computer-use agents (CUAs) driven by vision-language models, releasing annotation tools, a large-scale dataset (AgentNet: 22.6K desktop trajectories), and trained models. The key training insight is that augmenting state-action pairs with reflective long chain-of-thought reasoning is necessary for performance to scale with data. OpenCUA-72B achieves 45.0% on OSWorld-Verified, the new open-source SOTA, surpassing proprietary OpenAI CUA and matching Claude 4 Sonnet.

Problem

State-of-the-art CUA systems (Anthropic, OpenAI) remain closed—training data, architectures, and recipes are proprietary. Existing open-source GUI datasets are narrow in scope (mobile-only or web-only), lack scale, and prior works omit sufficient modeling details for replication. Direct SFT on raw human demonstrations yields only 4.4% success on OSWorld even with ~27K trajectories, indicating that data alone is insufficient without the right reasoning augmentation.

Method

OpenCUA comprises three integrated components:

AgentNet Tool & Dataset: A cross-OS annotation app captures natural human desktop interactions (screenshots + keyboard/mouse events) without disrupting workflow. This yields 22,625 trajectories (avg. 18.6 steps) across Windows, macOS, and Ubuntu, covering 140+ apps and 190+ websites at resolutions up to 4K.

Reflective Long CoT Synthesis: Raw demonstrations are converted to state-action pairs via an action discretization pipeline, then augmented with structured three-level reasoning: L3 (observation/scene description), L2 (reflective thought: state-transition analysis, error detection, planning), L1 (executable action). A reflector LLM compares before/after screenshots to flag incorrect or redundant steps; a generator then produces full CoT conditioned on reflection output and action history. claude-3-7-sonnet-20250219 is used as the synthesis backbone.

Modeling Recipe: Pure vision-based agents consume screenshots (no AXTree/DOM). History is encoded as a dialogue-style inner monologue (textual) plus 3 recent screenshots (visual). Training mixes L1/L2/L3 CoT levels, GUI grounding data (ShowUI, UGround, AXTree bounding boxes), platform demonstrations, and general SFT data (math, OCR, long-context). At inference, L2 CoT is used to maximize reflective reasoning. SFT is applied to Qwen2.5-VL (7B/32B/72B), Qwen2-VL-7B, and Kimi-VL-A3B.

Key Contributions

  • AgentNet Tool: cross-OS (Windows/macOS/Ubuntu) annotation infrastructure capturing personalized desktop environments with video, screenshots, and action flows
  • AgentNet Dataset: first large-scale desktop trajectory dataset with inner monologue annotations (22.6K tasks, 18.6 avg steps, 3 OSes, 140+ apps)
  • AgentNet Bench: offline evaluation benchmark (100 held-out tasks) correlated with OSWorld online metrics, enabling faster iteration
  • Reflective CoT synthesis pipeline (reflector → generator → summarizer) that injects error-correction reasoning into training trajectories
  • OpenCUA-72B: new open-source SOTA on OSWorld-Verified (45.0% at 100 steps) and UI-Vision grounding (37.3%)
  • Empirical demonstration of data scaling laws for CUAs and test-time compute scaling via Pass@n

Results

  • OSWorld-Verified (100 steps): OpenCUA-72B: 45.0% vs. UI-TARS-72B-DPO: 27.1%, Qwen2.5-VL-72B: 9.7% (base), Claude 4 Sonnet: 43.9%, Claude Sonnet 4.5: 61.4%
  • OSWorld-Verified Pass@3: OpenCUA-72B improves from 45.0% → 53.2%; OpenCUA-32B from 34.2% → 45.6%
  • AgentNet Bench (Avg SR): OpenCUA-32B: 79.1% vs. OpenAI CUA: 73.1%, Qwen2.5-VL-72B: 67.0%
  • GUI Grounding – UI-Vision: OpenCUA-72B: 37.3% (SOTA); ScreenSpot-Pro: 60.8%
  • Data scaling: Increasing Ubuntu data 3K→10K yields +72% avg. performance; Win/Mac 3K→14K yields +125%
  • Cross-domain benefit: Adding 14K Win/Mac data to 7K Ubuntu training improves OSWorld from 9.8% → 18.5%
  • Base recipe without CoT: only 4.4% on OSWorld; reflective CoT is the primary performance driver (confirmed by ablations)
  • Pass@n upper bound (OpenCUA-Qwen2-7B, 50 steps): Pass@1: 18.4% → Pass@16: 39.2% (+113%)

Limitations

  • OpenCUA-72B still trails Claude Sonnet 4.5 by ~16 percentage points at 100 steps; proprietary models maintain a significant gap
  • Performance gains from extending step budget from 50→100 steps are marginal, indicating poor self-error-recognition and loop-avoidance
  • Models exhibit platform-specific domain gaps: Ubuntu-trained models favor OSWorld; Windows/macOS-trained models favor WindowsAgentArena
  • AgentNet Bench, while correlated with online metrics, shows elevated scores for fine-tuned models due to domain and action-space alignment
  • CoT synthesis relies on claude-3-7-sonnet as the backbone reflector/generator, introducing a dependency on a proprietary model for training data creation
  • Only 5K of the Ubuntu subset (annotated by Moonshot AI) is publicly released; full dataset availability appears partial

Relevance to Vision-Language Models

OpenCUA directly demonstrates how VLMs can be specialized into grounded GUI agents through structured CoT fine-tuning, establishing a clear recipe: scale diverse screenshot-action trajectory data, augment with reflective inner monologue, and mix general vision-language SFT. The finding that raw SFT on state-action pairs yields near-zero performance until reflective reasoning is added is a strong result for the VLM-as-agent paradigm—it shows that agentic capability is not latent in base VLMs but must be explicitly injected via reasoning supervision. The test-time scaling results (Pass@n) also position VLM-based CUAs as candidates for inference-time compute scaling, connecting to broader trends in reasoning-augmented VLMs.