Skip to content

EvoGUI: An Evolution-Aware Benchmark for GUI State-Transition Understanding

🕒 Published (v1): 2026-07-19 03:29 UTC · Source: Arxiv · link

Why this paper was selected

Jiwen Lu (Tsinghua); EvoGUI isolates state-transition reasoning from end-to-end GUI success

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

EvoGUI is a diagnostic framework that mechanically converts normalized GUI interaction trajectories into three VQA probes—temporal ordering, inverse action/value prediction, and contrastive one-step successor discrimination—requiring no additional task-label annotation. Instantiated as EvoGUI-Bench (3,000 instances, 120 domains, from Mind2Web and WebLINX), it reveals that the best of 28 evaluated VLM configurations reaches only 60.4 EvoGain, and neither model scale nor GUI specialization reliably predicts transition-level performance.

Problem

End-to-end GUI-agent benchmarks report aggregate task-success rates that conflate perception, OCR, grounding, planning, tool use, and error recovery, making it impossible to isolate whether a model understands how actions transform interface states. No scalable, annotation-free method existed to produce transition-level diagnostic probes from existing trajectory logs, and prior dynamics-oriented evaluations (ENACT, MobileWorldBench) did not address the GUI setting with explicit perceptual and counterfactual bounds.

Method

EvoGUI normalizes GUI interaction logs into a unified trajectory schema \(\tau = [(s_t, a_t, v_t, s_{t+1}, m_t)]_{t=1}^{T}\) (consecutive screenshots, recorded action, optional value, metadata), then mechanically mines three VQA probes:

  • T1 – Temporal Ordering: given \(K \in \{3,4,5\}\) screenshots from the same trajectory in shuffled order, predict the chronological permutation; scored by exact-match ordering accuracy.
  • T2 – Inverse Action/Value Prediction: given \((s_t, s_{t+1})\), predict the normalized action class \(\in \{\text{CLICK, TYPE, SELECT}\}\) (action-only accuracy) and additionally the exact logged value for TYPE/SELECT (joint accuracy).
  • T3 – One-Step Reachability Discrimination: given \(s_t\) and two shuffled candidates, identify the logged adjacent successor \(s_{t+1}\) against a distractor drawn from one of three tiers—cross-trajectory, same-domain, or long-skip (\(s_{t+n},\, n \ge 3\)); scored by pairwise accuracy.

All labels are derived from trajectory adjacency and recorded actions; human inspection filters visually ambiguous T3 pairs (async page loads, ads, incidental refreshes) without assigning new labels. The composite EvoGain Index normalizes each task score against its random-choice baseline: $\(\text{NormScore}_i = \frac{\text{ModelScore}_i - \text{Baseline}_i}{100 - \text{Baseline}_i} \times 100\)$ and EvoGain is the macro-average over T1, T2-joint, and T3.

Key Contributions

  • EvoGUI framework: annotation-free pipeline converting normalized GUI trajectories into T1/T2/T3 VQA diagnostics; extensible to mobile/desktop via action-vocabulary expansion.
  • EvoGUI-Bench: 3,000-instance benchmark across 120 web domains (Mind2Web + WebLINX), balanced over T1 sequence length and T3 distractor tier.
  • EvoGain Index: composite metric that normalizes task scores against per-task random baselines and aggregates across structurally different probes.
  • Evaluation of 28 VLM configurations zero-shot, covering both open-weight and closed-weight models spanning sub-1B to 400B-class scales.
  • Diagnostic validity controls: black-image and metadata-only ablations confirming material visual dependence; indicative correlation with OSWorld end-to-end success (\(\rho = 0.90\) Spearman over 5 overlapping models).

Results

  • Best model: Gemini-3-Flash-Preview at 60.4 EvoGain; second: Seed-2.0-lite-260428 at 50.7.
  • Random-choice baseline: 0.0 EvoGain (T1 = 8.0%, T2-action = 33.3%, T2-joint = 15.0%, T3 = 49.9%).
  • Even the best model exceeds random by only +50.3 pts on T1, +50.4 pts on T2-joint, +33.7 pts on T3, indicating substantial headroom across all probes.
  • GUI-specialized UI-TARS-1.5-7B reaches only 25.4 EvoGain, below many general-purpose models.
  • Scale is non-monotonic: Qwen3.5-397B-A17B (39.5) scores below Qwen3.5-122B-A10B (43.8); Qwen3-VL-235B-A22B (37.5) below Qwen3.5-35B-A3B (40.5).
  • T2 action-only vs. joint ranking is nearly identical (Pearson \(r = 0.991\)), ruling out exact-value recovery as the sole driver.
  • Removing visual input in the Qwen3.5-27B control collapses T1 accuracy from 28.1% to ~7%, T3 from 70.8% to ~49% (near chance), confirming screenshot dependence.

Limitations

  • OSWorld correlation is based on only five overlapping models with partially mixed reporting protocols; association is indicative, not predictive.
  • Current instantiation is browser-centered (Mind2Web + WebLINX); mobile/desktop extension requires action-vocabulary expansion and domain-specific quality auditing.
  • T2 joint accuracy conflates causal transition understanding with OCR and value-binding ability, especially for TYPE and SELECT actions.
  • T3 distractors are sampled from logs rather than execution-validated counterfactuals; the task measures discrimination against the sampled candidate, not universal one-step unreachability.
  • Benchmark quality depends on the coverage and fidelity of the source human/expert trajectories; only 3,000 instances, 49% from WebLINX.
  • Larger-scale human audits and executable validation of counterfactuals are absent.

Relevance to Vision-Language Models

EvoGUI directly targets a blind spot in VLM evaluation: standard image-understanding and end-to-end agent benchmarks do not isolate whether a VLM can represent action-conditioned state transitions, a capability critical for GUI agents. The finding that model scale and GUI-specialization training do not reliably predict EvoGain challenges the assumption that stronger general VLMs are automatically better GUI-world modelers, motivating transition-level probes as a distinct evaluation axis. The annotation-free construction pipeline makes it straightforward to extend to new trajectory corpora, enabling the community to track this capability as VLMs evolve. For researchers studying grounding, world modeling, or agentic VLMs, EvoGUI-Bench offers a controlled diagnostic that decouples perceptual state-change understanding from planning and execution.