Skip to content

CUA-Suite: Massive Human-annotated Video Demonstrations for Computer-Use Agents

🕒 Published (v1): 2026-03-25 15:52 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CUA-Suite is a large-scale ecosystem of human-annotated expert video demonstrations for desktop computer-use agents, comprising three unified resources: VideoCUA (55 hours / 6M frames of 30 fps screen recordings across 87 applications), GroundCUA (56K annotated screenshots with 3.6M UI-element bounding boxes), and UI-Vision (a 450-task benchmark). Preliminary evaluation shows current foundation action models fail on roughly 60% of professional desktop tasks, and even the best grounding model (MAI-UI-32B) achieves only 47.7% average accuracy on UI-Vision.

Problem

CUA training is bottlenecked by the absence of continuous, high-quality human demonstration data at scale. The largest prior open dataset (ScaleCUA) provides only ~2M screenshots (~18.5 hours equivalent at 30 fps), capturing sparse click coordinates but discarding intermediate visual feedback, kinematic cursor dynamics, and temporal continuity needed for continuous spatial control, visual world models, and long-horizon planning. Human-curated datasets that do exist either target web/mobile (not professional desktop) or cover only one modality (grounding without temporal context, or video without frame-level element annotations).

Method

Data engine (shared foundation): Human annotators record continuous 30 fps screen video while executing expert-designed, real-work tasks across 87 open-source desktop applications in 12 categories. Every mouse event, keystroke, and scroll is logged with millisecond precision, yielding synchronized kinematic cursor traces.

Dense annotation pipeline: 1. Keyframes immediately preceding state-changing actions are extracted. 2. Annotators manually label every visible UI element with bounding boxes and textual labels; OCR (PaddleOCR) supplements long-text regions. 3. ~50% of elements are classified into 8 functional semantic categories. 4. Multi-layered reasoning annotations (Observation → Thought → Reflection → Action) average 497 words per step. 5. Expert human review enforces quality assurance at each stage.

Three derived resources from this single engine: - VideoCUA — continuous video trajectories for agentic execution training. - GroundCUA — 700K instruction-tuning examples derived from 56K densely annotated screenshots, used to train GroundNext (3B/7B) via SFT + RL. - UI-Vision — 450-task benchmark testing Element Grounding, Layout Grounding, and Action Prediction.

Key Contributions

  • VideoCUA: ~10,000 human-demonstrated tasks, 87 applications, ~55 hours, 6M frames at 30 fps — the largest open expert video corpus for desktop CUA, \(>\)2.5\(\times\) ScaleCUA by video hours.
  • GroundCUA: 56K annotated screenshots, 3.6M element annotations, yielding 700K instruction-tuning pairs; trains GroundNext-3B/7B achieving SoTA grounding results.
  • UI-Vision benchmark: Multi-axis evaluation (element grounding, layout grounding, action prediction) across 87 professional desktop applications with dense expert-verified ground truth.
  • Unified data engine: Single human-annotation pipeline that is the shared source of truth for all three resources, ensuring causal consistency (every element labeled, every action logged).
  • Fully open-source release of data, benchmarks, and models.

Results

  • UI-Vision element grounding (Table 1): Best model MAI-UI-32B achieves 47.7% average (Basic: 59.1%, Functional: 57.1%, Spatial: 26.9%); prior SoTA UI-TARS-72B achieves 25.5%. Performance has roughly doubled year-over-year.
  • Spatial grounding remains the hardest axis across all models; top models cluster well below 30% on the Spatial split.
  • Scaling effect: OpenCUA improves +7.6 absolute points from 7B→72B; PhiGround-7B paired with o3 planner improves +9.0 points (27.2%→36.2%).
  • GroundNext-3B + o3 planner achieves 50.6 on OS-World Verified.
  • Action prediction / task completion: current foundation action models fail on approximately 60% of professional desktop tasks.

Limitations

  • Coverage is restricted to 87 open-source applications; closed-source or proprietary enterprise software is absent.
  • ~50% of elements receive semantic category labels; the remaining 50% lack functional classification.
  • Action Prediction detailed results are deferred to the companion paper (Nayak et al., 2025) rather than reported in full here.
  • Annotation pipeline relies on human labor, limiting scalability compared to synthetic approaches.
  • Format compatibility with OpenCUA/ScaleCUA is demonstrated but fine-tuning experiments comparing video-based vs. screenshot-based training signals are described as future work rather than completed experiments.
  • No ablation on the impact of kinematic cursor traces in isolation from other annotation layers.

Relevance to Harnesses / Meta-Harnesses

CUA-Suite is itself a data-collection meta-harness: a unified pipeline that ingests raw human interaction (screen video + action logs), applies layered annotation operators, and emits multiple downstream-ready resources (training sets, instruction-tuning corpora, evaluation benchmarks) from a single source of truth. This is structurally analogous to a meta-harness that wires together heterogeneous tools — here, the "tools" are the annotation, OCR, QA, and derivation stages. For researchers building agent harnesses, CUA-Suite supplies the dense, causally consistent supervision signal (continuous video + grounding + CoT) that prior evaluation harnesses (OSWorld, WebArena) lack, enabling harnesses that go beyond sparse reward to video-based reward modeling and world-model lookahead. The format compatibility layer — VideoCUA is explicitly designed to plug into OpenCUA and ScaleCUA training pipelines — demonstrates the meta-harness design principle of interchangeable, composable data modules.