Skip to content

ProBench: Benchmarking GUI Agents with Accurate Process Information

🕒 Published (v1): 2025-11-12 09:49 UTC · Source: Arxiv · Venue: AAAI 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ProBench is a mobile GUI agent benchmark with 200+ tasks that exposes a critical flaw in existing evaluations: judging task success solely from the final screen state masks failures when intermediate process steps are skipped. It introduces a Process Provider — combining accessibility-tree parsing and MLLM-based screenshot comparison — to automatically supply accurate process information for evaluation. Across all tested models, even the best (Gemini 2.5 Pro) achieves only 40.1% overall accuracy.

Problem

Existing dynamic GUI benchmarks (AndroidWorld, AndroidLab, B-MoCA) evaluate task completion exclusively via the final screen state, but many real-world tasks require verifying critical intermediate steps (e.g., "sort by price before selecting"). This produces false positives: an agent that skips a required sorting step still reaches a visually identical purchase confirmation screen. Prior attempts to incorporate process (SPA-Bench, A³) either rely on rigid pre-defined trajectories that suppress planning evaluation, or use LLM decomposition with inherently noisy accuracy.

Method

ProBench is implemented over Android Debug Bridge (adbutils) against 34 bilingual apps (14 English, 20 Chinese) spanning shopping, social, travel, media, and system categories. Tasks are divided into two types: - State-related Tasks (ST): completion determinable from the final screenshot alone. - Process-related Tasks (PT): require verifying both the final state and specific intermediate operations.

The Process Provider supplies process information via two optional components: 1. Structure Description Converter: after each CLICK, parses the accessibility tree, locates the smallest enclosing clickable node at the click coordinates, and extracts text/content-desc/resource-id as a compact action description. 2. MLLM-based Summarizer: merges before/after screenshots with the click coordinate marked, then prompts an MLLM to generate a natural-language summary of what changed.

A Judger (Gemini 2.5 Pro in experiments) receives the final screenshot plus the full textualized action sequence to make a binary Success/Failure/Uncompleted determination. Evaluation accuracy against human annotation: 96.0% for ST (evaluator only), 89.7% for PT + Structure Description Converter, 94.1% for PT + MLLM-based Summarizer.

Key Contributions

  • ProBench: 217 tasks across 34 mainstream bilingual apps, both State-related and Process-related task types.
  • Process Provider with two complementary automatic process-information extraction methods (a11y-tree-based and MLLM-based), verified against human annotation at ≥89.7% accuracy.
  • Systematic evaluation of 10 agents (proprietary: GPT-4o, Claude 4 Sonnet, Gemini 2.5 Pro; open-source: Qwen2.5-VL 7B/32B/72B, InternVL3-8B; GUI-specific: UI-TARS-1.5-7B, UI-R1-E-3B, GUI-R1-3B).
  • Error taxonomy identifying three universal failure modes: insufficient grounding, insensitivity to historical operations, and oversimplified task planning.

Results

  • Best overall: Gemini 2.5 Pro at 40.1% (45.6% ST, 27.9% PT).
  • Best open-source: Qwen2.5-VL-72B at 36.9% overall; 63.5% on English ST, 30.4% on English PT.
  • ST vs PT gap is universal: every model scores higher on ST than PT; the gap for Qwen2.5-VL-72B is 40.9% ST vs 27.9% PT overall.
  • Only Qwen2.5-VL-72B exceeds 60% on any single subtask (English ST: 63.5%).
  • GUI-specific models underperform large general models: UI-TARS-1.5-7B reaches 8.8% overall; UI-R1-E-3B scores 0.0% across all settings (missing COMPLETE action in training).
  • Strong scaling: Qwen2.5-VL 7B→32B→72B yields 5.1%→16.6%→36.9% overall.
  • Agents perform best on production/system apps and worst on social/lifestyle apps.
  • Early-stop rate (repetitive action loops) is high across all models: 49.6% of Gemini's uncompleted tasks, up to 77.9% for InternVL3-8B.

Limitations

  • Binary Success/Failure metric cannot capture degrees of partial progress.
  • Tasks capped at 15 steps; longer real-world workflows are excluded.
  • Process-related evaluation accuracy is 89.7–94.1%, not perfect — some process information is still noisy or missing due to app-level a11y tree variance.
  • Chinese app evaluation requires a physical device, limiting reproducibility.
  • Task set (217 tasks) is relatively small, concentrated on Chinese/English mobile apps; coverage of desktop GUIs, web browsers, or other OS environments is absent.

Relevance to Agentic AI / LLM Agents

ProBench directly addresses the measurement problem for multi-step GUI agents: if evaluation only checks the end state, agents that skip critical intermediate reasoning steps appear more capable than they are, obscuring real deficiencies. The finding that Process-related Tasks reliably expose a ~10–20% accuracy gap relative to State-related Tasks across all model families quantifies how much current benchmarks over-credit agent performance. The identified failure modes — poor GUI grounding in frontier proprietary models, inability to track action history, and flat task planning — map directly onto the core capability requirements for agentic LLM systems operating in real environments. The automatic Process Provider design is a reusable infrastructure pattern for any benchmark involving sequential action evaluation.