WindowsWorld: A Process-Centric Benchmark of Autonomous GUI Agents in Professional Cross-Application Environments

🕒 Published (v1): 2026-04-30 12:13 UTC · Source: Arxiv · link

Why this paper was selected

WindowsWorld: process-centric benchmark for GUI agents across professional cross-app tasks

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

WindowsWorld is a process-centric benchmark of 181 tasks across 17 Windows desktop applications, with 77.9% requiring cross-application coordination drawn from 16 professional personas. It replaces binary success/failure scoring with per-checkpoint intermediate evaluation (avg. 4.97 checkpoints/task). State-of-the-art agents achieve at most ~20% final success rate on multi-application tasks, revealing a fundamental gap between single-app and professional-grade workloads.

Problem

Existing GUI agent benchmarks (OSWorld, AndroidWorld, Windows Agent Arena) are dominated by single-application tasks (<28% multi-app) and rely on binary "all-or-nothing" success-rate metrics. This makes it impossible to distinguish partial progress from total failure on long-horizon tasks, and fails to capture the cross-application coordination central to real professional workflows. Benchmark construction is also bottlenecked by labor-intensive manual curation that does not scale.

Method

Task generation uses a four-stage human-in-the-loop multi-agent pipeline: 1. Generator Agent — LLM (DeepSeek-V3.2) produces persona-grounded tasks for 16 professional roles (accountant, software engineer, etc.) with real URLs from allowlisted open platforms. 2. Refiner — a 4-node automated pipeline: (i) semantic deduplication via cosine-similarity thresholding (\(\tau=0.85\)), (ii) async HTTP validity auditing, (iii) LLM-based dependency reasoning converting procedural preconditions into declarative environment states, and (iv) metric refinement standardizing checkpoint criteria. 3. Human Reviewer — four annotators reject ambiguous instructions, subjective criteria, or proprietary-software dependencies. 4. Environment Generator — synthesizes required files (.xlsx, .docx, .py, etc.) with Smart File Merging across same-persona tasks for consistent data resources.

Evaluation formalizes GUI interaction as a POMDP and defines two metrics using a VLM judge (Qwen3-VL-Plus): $\(S_{\text{int}} = \frac{1}{|C|}\sum_{k=1}^{|C|} J(\tau, c_k), \quad S_{\text{final}} = J(\tau, s_T)\)$ where \(C\) is the set of intermediate checkpoints and \(s_T\) is the terminal state. Checkpoints are state-based (not trajectory-specific), tolerating keyboard shortcuts, menu alternatives, etc.

Key Contributions

• 181-task Windows desktop benchmark with 77.9% multi-application tasks spanning 17 apps and 16 professional personas across 5 domains
• Four-level difficulty taxonomy: L1 (single-app atomic), L2 (multi-app linear), L3 (dynamic conditional reasoning across ≥3 apps), L4 (intentionally infeasible tasks for goal-rejection evaluation)
• Process-aware evaluation with avg. 4.97 intermediate checkpoints per task, replacing binary scoring
• Scalable human-in-the-loop multi-agent pipeline for benchmark construction, including LLM-driven environment file synthesis
• Near-human VLM judge reliability: Pearson \(r=0.9108\) (\(S_{\text{int}}\)), Cohen's \(\kappa=0.8668\) at checkpoint level

Results

• Best overall model (Gemini-3-flash-preview, Hybrid input): \(S_{\text{int}}=50.32\%\), \(S_{\text{final}}=20.44\%\) — a ~30 pp efficiency-completion gap indicating agents complete sub-goals but fail to synthesize terminal states
• Single-app vs. multi-app success rate drop: OSWorld 77.9% → WindowsWorld multi-app 22.1% (Agent-S3 w/ GPT-5); AndroidWorld 90.5% → 27.4% (GPT-4V)
• Gemini-3-flash \(S_{\text{final}}\) by level: L1=35.90%, L2=17.50%, L3=8.00% (Screenshot); demonstrates steep horizon-length degradation
• L4 infeasible task rejection: GPT-5.2 (SoM) achieves only 25% — best among pure models; UIPath scores high on L4 only because it frequently declares failure by default
• Agent-S3 w/ Gemini-3-flash (best agent): \(S_{\text{int}}=42.42\%\), \(S_{\text{final}}=17.13\%\)
• Hybrid (Screenshot + Accessibility Tree) input consistently outperforms Screenshot-only and Set-of-Marks for most models; Gemini-3-pro gains +7.9 pp \(S_{\text{int}}\) with accessibility trees

Limitations

• Only 181 tasks — small absolute size limits statistical power for fine-grained per-application analysis
• Restricted to Windows OS and a fixed set of 17 applications; no Linux/macOS coverage
• VLM judge (Qwen3-VL-Plus) introduces false positives/negatives at complex visual states, despite high aggregate agreement
• Benchmark reflects professional personas from a particular cultural/occupational framing; may underrepresent non-Western workflows
• No automated reset for mid-task environment corruption; failures may cascade across steps

Relevance to Harnesses / Meta-Harnesses

WindowsWorld's multi-agent benchmark construction pipeline — Generator → Refiner → Human Reviewer → Environment Generator — is itself a meta-harness: a structured orchestration of LLM agents, automated validators, and human-in-the-loop checkpoints that produces evaluation infrastructure rather than task outputs. The Smart File Merging strategy and dependency-reasoning node demonstrate how a harness can maintain cross-task consistency at scale. For researchers building evaluation harnesses, the four-node Refiner pipeline (dedup, validity audit, dependency reasoning, metric refinement) is a directly applicable pattern for automated quality control of LLM-generated task specifications.