Skip to content

Grounding Computer Use Agents on Human Demonstrations

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Desktop grounding dataset for computer-use agents; bridges web and mobile gap

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

GroundCUA is a large-scale desktop UI grounding dataset of 3.56M human-verified element annotations across 87 applications, used to train the GroundNext family of VLMs. GroundNext (3B and 7B) achieves state-of-the-art grounding accuracy across five benchmarks using only 700K SFT samples—less than one-tenth the training data of prior SOTA—then further improves with RL post-training. The work demonstrates that expert-quality, densely annotated data is a stronger lever than raw scale for training reliable computer-use agents.

Problem

Desktop grounding—mapping natural language instructions to precise on-screen UI elements—is the critical bottleneck for computer-use agents. Existing datasets rely on automated pipelines (accessibility-tree traversal, HTML/DOM scraping, synthetic screen generation) that underrepresent small desktop controls (icons, toolbar buttons), produce incomplete or noisy labels, cover limited resolution ranges, and average only 7–14 annotated elements per screenshot versus the 64 dense annotations per image achievable by human experts. No large, human-verified desktop grounding dataset existed prior to this work.

Method

Dataset construction (GroundCUA): Expert annotators performed 10,000+ real computer-use task demonstrations across 87 open-source desktop applications. Keyframes captured immediately before user actions were densely labeled: every visible UI element received a bounding box, a textual label, and (for ~50% of elements) a category tag from eight classes. PaddleOCR extracted raw text for long on-screen strings. A multimodal LLM then generated three instruction types per element—Direct (attribute/positional), Functional (action-intent), and Spatial (relative-position)—grounded in the bounding box metadata and application context. This produced 700K SFT samples and 10K held-out RL samples.

Model training (GroundNext): Base model is Qwen2.5-VL-Instruct at 3B and 7B scales, with both the vision encoder and language model jointly fine-tuned. - SFT stage: Standard supervised fine-tuning on 700K instruction–screenshot pairs (8×H100, global batch 128). - RL stage: RLOO (Relative Leave-One-Out) policy optimization on 10K fresh elements not seen in SFT. The reward is a six-level discrete function of normalized signed distance \(D_\text{norm}\) between the predicted point \(\hat{p}\) and ground-truth bounding box \(B\):

\[D_\text{norm} = \frac{D(\hat{p}, B)}{D_\text{ref}}, \quad D_\text{ref} = \begin{cases} \tfrac{\text{diam}(B)}{2} & \hat{p} \in B \\ D_\text{max}(B, I) & \text{otherwise} \end{cases}\]

Scores range from \(-1.0\) (far outside) to \(+1.0\) (well inside, near center), with graduated intermediate penalties. No separate critic model is trained (RLOO avoids this). Group size \(n=8\), batch size 64, one epoch.

Key Contributions

  • GroundCUA dataset: 56K screenshots, 3.56M human-verified annotations, 87 applications, 12 categories; average 64 elements/screenshot (vs. ~8 for prior desktop sets), resolution range 0.39–7.0 MP, average element area 0.13% of screen (the smallest among compared datasets), permissive license.
  • GroundNext model series: 3B and 7B VLMs trained with SFT + RL achieving SOTA on desktop grounding benchmarks with ≤700K training samples vs. up to 9M in prior work.
  • RL reward design: Discrete normalized-distance reward that grades near-miss and far-miss errors differentially, without reward-model judges or auxiliary objectives.
  • Cross-platform generalization: Desktop-only training generalizes to mobile and web benchmarks (MMBench-GUI, ScreenSpot-v2).
  • Agentic evaluation: GroundNext used as the grounding module with o3 as a planner on OSWorld-Verified, surpassing larger models including OpenCUA-72B.

Results

SFT-only (five benchmarks: ScreenSpotPro, OSWorld-G, MMBench-GUI-L2, ScreenSpot-v2, UI-Vision): - GroundNext-3B (SFT): 66.4 avg vs. 54.3 (GUI-Actor-3B, next best); +12.1 pts with UI-V, +5.4 pts without UI-V. - GroundNext-7B (SFT): 69.2 avg (71.8 without UI-V) vs. 56.1 (JEDI-7B, next best); +13.1 pts with UI-V. - UI-Vision specifically: GroundNext-3B 58.2 vs. JEDI-3B 18.7; GroundNext-7B 58.7 vs. JEDI-7B 24.8. - OSWorld-G: GroundNext-7B 67.2 vs. JEDI-7B 54.1.

RL post-training: - GroundNext-7B (RL): 73.0 avg (without UI-V), best among all RL-tuned systems. - GroundNext-3B (RL): outperforms all 3B RL-tuned baselines; competitive with JEDI-7B.

Agentic evaluation (OSWorld-Verified, GroundNext + o3 planner): - GroundNext-3B surpasses OpenCUA-72B and multiple proprietary APIs; comparable to JEDI-7B despite 3B vs. 7B scale.

Data efficiency: GroundNext trained on 700K samples outperforms JEDI trained on 9M samples (>12Ă— fewer data points).

Limitations

  • Dataset covers only open-source desktop applications; closed-source software (Microsoft Office, Adobe suite) is not represented.
  • UI-Vision benchmark overlaps with GroundCUA's application coverage, making it a partially in-domain evaluation despite efforts to minimize annotation overlap.
  • RL stage uses only 10K samples on a single epoch; further scaling of RL training was not explored.
  • Instruction generation relies on a multimodal LLM, which may introduce systematic biases in the diversity of generated queries.
  • No evaluation of full end-to-end agent task success beyond OSWorld-Verified (with o3 as planner); grounding accuracy on static benchmarks may not fully predict agent performance on long-horizon tasks.
  • Annotation density and quality depend on the labeling company's annotator pool; inter-annotator agreement statistics are not reported in the main paper.

Relevance to Agentic AI / LLM Agents

Grounding—the mapping from a planner's symbolic action to a precise pixel-level target—is the last-mile bottleneck separating high-level LLM reasoning from reliable computer-use execution; this work directly addresses that gap with a purpose-built dataset and model family. The two-stage SFT→RL training recipe, including the normalized-distance discrete reward without a critic or reward model, is directly transferable to other agentic grounding tasks and shows that policy refinement can be achieved cheaply on small held-out data. The OSWorld-Verified results demonstrate a modular agent architecture (LLM planner + specialist grounding model) that outperforms monolithic large models, offering a practical blueprint for scalable computer-use agents. The data-efficiency finding—700K expert-annotated samples beating 9M automated samples—has broad implications for agentic dataset curation strategy.