Skip to content

Computer Agent Arena: Toward Human-Centric Evaluation and Analysis of Computer-Use Agents

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Human-centric evaluation paradigm for computer-use agents beyond static benchmarks

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Computer Agent Arena is an open-source, crowd-sourced evaluation platform for Computer-Use Agents (CUAs) that collects pairwise human preferences over agents executing real user tasks in cloud-hosted virtual machines. Across 2,201 votes from 1,058 users evaluating 12 agents, the resulting leaderboard diverges sharply from static benchmarks like OSWorld, exposing fundamental gaps in how scripted suites assess real-world agent capability.

Problem

Static CUA benchmarks (OSWorld, WebArena, WebVoyager, Online-Mind2Web) suffer from narrow task domains prone to overfitting/contamination, manually designed reward functions that miss process quality, no personalization or safety signals, and inability to capture real-world dynamism. They ignore how humans actually judge agents—by execution quality, clarification behavior, and error recovery—not only final-state correctness.

Method

The platform deploys cloud-hosted AWS EC2 VMs (based on OSWorld's AMI) with 600+ distinct desktop initializations spanning mainstream apps and websites. Two anonymized CUAs execute each user-submitted task in parallel on identically seeded VMs; users observe synchronized trajectory replays and submit pairwise preferences plus structured labels (correctness, safety, efficiency, step-wise grounding errors). Rankings are aggregated via the Bradley–Terry model:

\[\Pr(m_L \succ m_R) = \frac{\exp(\beta_{m_L})}{\exp(\beta_{m_L}) + \exp(\beta_{m_R})}\]

Elo scores are derived as \(E_m = 400\log_{10}(e^{\beta_m}) + 1000\), with 95% bootstrap confidence intervals used for final ranking. Agents also have access to a CALL_USER action for clarification queries, enabling measurement of human–agent interaction quality.

Key Contributions

  • Open-source crowd-evaluation platform with 600+ realistic VM initializations, parallel agent execution, and trajectory replay for non-technical raters.
  • A 2,201-vote, 12-agent preference dataset with Krippendorff's \(\alpha = 0.72\) (preference), \(0.78\) (correctness), \(0.68\) (safety) inter-annotator agreement.
  • Empirical demonstration of leaderboard rank inversions versus OSWorld/WebArena/WebVoyager for the same models.
  • Preference analysis showing correctness dominates, but moderate CALL_USER querying (1–2 times) yields an inverted-U win-rate boost while excessive querying hurts.
  • Error taxonomy exposing four failure modes underexposed by static benchmarks: tool-selection/use errors, long-horizon memory lapses and plan drift, insufficient information seeking, and fine-grained grounding/action-precision errors.
  • Empirical contrast of pure GUI agents vs. tool-integrated agents (e.g., CoAct-1 achieves 60.1% on OSWorld-Verified but ranks 5th on the Arena).

Results

  • Leaderboard top-4: Claude Sonnet 4 (Elo 1167, 52.0% correct) > Claude 3.7 Sonnet (1140, 52.3%) > UI-TARS-1.5 (1092, 49.9%) > Operator (1064, 37.4%).
  • General-purpose models underperform: GPT-5 ranks 8th (Elo 1002, 34.3%), Gemini 2.5 Pro last (829, 11.8%), despite strong multimodal capability.
  • Rank inversions vs. static benchmarks: Several OSWorld top performers are inverted; CoAct-1 (state-of-the-art on OSWorld-Verified) ranks 5th overall.
  • In-domain vs. OOD ablation: Recomputing Elo on 1,000 tasks split by OSWorld overlap causes significant rank shifts (e.g., UI-TARS-1.5 rises on in-domain tasks).
  • Correctness vs. efficiency: Strong linear correlation between correctness and win rate; step count and latency show negligible effect when correctness is held constant.
  • Moderate querying: 1–2 CALL_USER queries correlates with highest win rates; 0 or 4+ queries lower win rate.
  • Statistical validation: Permutation tests confirm pairwise win-rate differences significant at \(p < 0.01\); Cohen's \(d > 0.5\); power \(> 0.9\) for \(\Delta\text{Elo} \approx 50\) with \(> 200\) votes per pair.

Limitations

  • Limited vote counts for CoAct-1, OpenCUA, and GPT-5 (\(\approx\)108–110 votes each); rankings for these models marked provisional.
  • Crowdsourced tasks may still skew toward tech-savvy users despite demographic diversity, potentially underrepresenting non-technical use cases.
  • Trajectory evaluation requires users to watch full replay videos, which may introduce fatigue bias for longer trajectories.
  • Platform currently limited to Windows/Ubuntu; no mobile or specialized OS coverage.
  • Tool-integrated agents' failures (e.g., silent API errors) are invisible in GUI replays, making error attribution harder for non-expert raters.
  • The paper's text is truncated; full error analysis details and some ablation results may not be fully represented above.

Relevance to Agentic AI / LLM Agents

This work directly addresses the evaluation bottleneck for long-horizon, interactive agents operating in real GUI environments—a core challenge as CUAs move toward deployment. The finding that correctness alone does not explain human preference, and that judicious CALL_USER behavior and self-correction matter, has direct implications for agent policy design, reward modeling, and RLHF pipelines targeting real-world computer use. The leaderboard inversion result—tool-integrated agents excel on scripted benchmarks but underperform on open-ended real-user tasks—is a critical warning for the community developing agentic architectures with code/API execution. The platform also provides an evaluation methodology directly applicable to any agent system that operates over a desktop environment, complementing offline benchmarks with continuous, contamination-resistant human signal.