Skip to content

GUI-Xplore: Empowering Generalizable GUI Agents with One Exploration

🕒 Published (v1): 2025-03-22 09:30 UTC · Source: Arxiv · Venue: CVPR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

GUI-Xplore introduces a dataset and agent framework built around a single pre-recorded exploration video per app, enabling GUI agents to generalize across unfamiliar applications and heterogeneous task types. The paired Xplore-Agent converts raw exploration video into a GUI Transition Graph via action-aware keyframe extraction and LVLM-based page clustering, then feeds that graph as structured context to an LLM for multi-task reasoning. Xplore-Agent achieves ~10% accuracy gain over SOTA GUI agents on cross-app automation.

Problem

Existing GUI agent datasets (Mind2Web, AITW, WebArena, LLaMATouch) share two fundamental gaps: (1) they ignore developer-induced structural variation across apps, so models fail to transfer interaction logic to unfamiliar software; (2) they restrict tasks to basic navigation/automation, preventing evaluation of the broader capability spectrum (environment understanding, temporal action reasoning). No prior dataset provides app-specific inductive priors at inference time without requiring costly live interaction.

Method

Data collection. 312 Android apps are explored once using DroidBot (207 apps, automated) or a Scrcpy-based manual tool (105 apps), producing 115 h of screen-recording video (avg. 23.73 min/app) plus view hierarchies (VH) and action annotations.

Five downstream tasks are formulated as 5-way multiple-choice QA, covering two axes: - Environment Understanding: Application Overview, Page Analysis, Application Usage - Operational Behavior Understanding: Action Recall, Action Sequence Verification

Xplore-Agent (two-stage pipeline): 1. Action-aware GUI Modeling — luminance difference (Y-Diff in YUV space) detects frame boundaries at action transitions; a fine-tuned QwenVL-7B generates simplified VH and action labels for each keyframe pair, producing a textual Exploration Sequence. 2. Graph-Guided Environment Reasoning — an LVLM clusters keyframes into screen nodes online (sequential decision: existing node vs. new node), building a GUI Transition Graph where nodes carry functional descriptions and edges carry action types. The node list + edge list are serialized into a prompt; GPT performs downstream Q&A over this context.

Key Contributions

  • GUI-Xplore dataset: 312 apps, 32,569 QA pairs across 5 hierarchical tasks, 115 h of exploration video, 34,367 VH captures, 41,293 action annotations; covers 33 subcategories in 6 domains.
  • Explore-then-reason paradigm: provides app-specific structural priors via a single task-agnostic exploration video rather than relying solely on generalized training-time knowledge.
  • Xplore-Agent baseline: action-aware keyframe extraction → VH/action generation → LVLM-based GUI clustering → Transition Graph → LLM Q&A; achieves 10% improvement over SOTA on cross-app automation.
  • Cross-task benchmark: reveals large performance discrepancies among current SOTA models across the five task types, establishing headroom for future work.

Results

  • Xplore-Agent vs. SOTA GUI agents (CogAgent, SeeClick, AUTO-UI) on Cross-App Automation test set (500 operations, 20 apps): ~10% absolute improvement in accuracy for previously unseen apps.
  • Benchmarking video-understanding models (VideoTree, CogVLM2-Video, VideoChat2, VideoLLaMA, GPT) and GUI agent methods across all five GUI-Xplore tasks shows significant performance discrepancies, indicating no current method achieves generalizable GUI agent behavior across all task types. (Specific per-task accuracy numbers are not included in the provided text excerpt.)

Limitations

  • The 10% gain leaves substantial headroom; the paper acknowledges "significant potential for further enhancement."
  • Automated DroidBot exploration (207/312 apps) does not replicate authentic user behavior, introducing coverage and quality differences vs. manual data.
  • Xplore-Agent uses GPT for clustering and reasoning stages, making it expensive and closed-source dependent.
  • Exploration videos average ~200 pages and ~24 min; long-video encoding remains a bottleneck for end-to-end models.
  • All tasks are cast as 5-way multiple-choice QA, which sidesteps open-ended generation evaluation.
  • Test set is restricted to the 20% manual-annotation split, potentially limiting domain diversity in evaluation.

Relevance to Harnesses / Meta-Harnesses

GUI-Xplore instantiates an exploration harness: a structured pipeline that wraps raw screen-recording input through modular stages (keyframe extraction → VH generation → action generation → graph clustering → LLM Q&A), functioning as a meta-harness that transforms unstructured video into structured agent-readable context before any task-specific module runs. The two-stage Xplore-Agent architecture — where Stage 1 produces intermediate representations consumed by Stage 2 — mirrors the scaffold-then-execute pattern common in meta-harnesses. The dataset's task-agnostic exploration design (one video serves all five downstream tasks) is directly analogous to meta-harness philosophy: a single generalized preprocessing pass amortized across multiple downstream consumers.