Skip to content

SEE: Structure-aware Exploring \& Exploiting for Long-horizon GUI Agent Trajectory Synthesis

🕒 Published (v1): 2026-07-20 15:16 UTC · Source: Arxiv · link

Why this paper was selected

GUI agent trajectory synthesis with VLMs; long-horizon coverage; agentic scaffolding

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SEE is a two-stage data synthesis framework for GUI agents that first builds an explicit screen–element transition graph via structure-aware exploration, then composes long-horizon trajectories by planning over that graph rather than re-running live agents. It produces a dataset with average trajectory length 14.8 steps and 24.63 interactive elements per screen, substantially exceeding prior public GUI datasets. Fine-tuning Qwen3-VL-4B on SEE-Train raises task success rate from 62.61% to 77.29% on the SEE-Test benchmark.

Problem

Existing GUI agent training data is dominated by short, "happy-path" interactions on simple, relatively static apps. Human demonstrations are expensive and hard to scale; on-policy synthesis pipelines over-sample common flows while missing rare transitions and long-horizon, multi-screen procedures. The result is a fundamental mismatch: modern mobile apps are visually crowded, dynamically updated, and require recovery from interruptions—yet no public dataset occupies the high-trajectory-length, high-screen-complexity quadrant needed to train robust agents for them.

Method

SEE operates in three phases over real mobile apps:

1. Structure-aware exploration. The agent parses candidate UI elements using a UI parser, then a VLM generates short semantic descriptions for ambiguous icons. Elements are ranked by a composite score \(s(e_{t,i}) = s_\text{sem}(e_{t,i}) + s_\text{lay}(e_{t,i}) - p_\text{hist}(e_{t,i})\) combining semantic alignment with app category, layout position priors (favoring top/bottom bars), and history-based penalties for redundant choices. Screen identity is determined via Hungarian matching on MobileNet visual embeddings and bounding-box positions across predefined stable regions: $\(C_{ij}(o, o') = \alpha\, d_\text{cos}(f_i, f_j') + \beta\, d_\text{pos}(b_i, b_j')\)$ This builds a bipartite graph \(G = (V, E)\) with screen nodes, element nodes, transition edges \((u, a, u')\), and containment edges \((u, e)\).

2. Reflection and refinement. After each action, a reflection step verifies whether the observed outcome is semantically consistent with the intended transition. Only verified edges are added to \(G\); element descriptions are refined based on observed outcomes. System interruptions (permission dialogs, overlays) are recorded with their recovery paths rather than discarded.

3. Graph-based synthesis. Candidate screen nodes are sampled from \(V\) (with emphasis on long-tail nodes), and an LLM generates an ordered subgoal list and high-level instruction conditioned on node semantics. BFS over \(G\) connects consecutive subgoal targets into a complete action sequence. Low-level step instructions are generated cheaply by conditioning on stored node/edge annotations rather than re-parsing raw screenshots, making marginal synthesis cost primarily a function of lightweight subgoal prompting.

Key Contributions

  • Structure-aware exploration algorithm that builds a screen–element transition graph without manual trajectory annotation, using semantic ranking, layout priors, and history penalties.
  • Graph-based synthesis pipeline that decouples structural acquisition from trajectory construction, enabling controllable length and coverage via subgoal count and graph path selection.
  • SEE dataset: 47K training + 5K test trajectories with average 14.8 steps, 24.63 elements/screen, and three-level aligned supervision (high-level instruction, ordered subgoals, step-level action descriptions); test apps are disjoint from training apps.
  • Reflection-based graph refinement that filters spurious edges and improves transition reliability before large-scale synthesis.

Results

  • Qwen3-VL-4B fine-tuned on SEE-Train achieves 77.29% SR on SEE-Test vs. 62.61% for the base model (+14.68 pp).
  • Per-step accuracy improvements (base → fine-tuned): Tap 54.76%→67.16%, Swipe 81.10%→97.20%, Back 85.43%→97.76%, Type 73.45%→96.67%.
  • Grounding accuracy: 60.79% (base) → 69.91% (fine-tuned); Action Type accuracy: 93.04% → 97.87%.
  • Adding retrieved context on top of fine-tuning yields marginal further gains (+0.67% SR, +1.94% grounding), suggesting the fine-tuned model already internalizes most structural knowledge.
  • SEE reaches 24.63 avg elements/screen vs. 13.59–17.26 for all compared datasets (AndroidLab, UI-Genie, AndroidControl, AMEX, GUI-Xplore, OS-Genesis).
  • Average trajectory length 14.8 steps vs. 5.5–12.8 for all compared public datasets.

Limitations

  • SEE-Test shows that fine-grained grounding (not action-type classification) remains the primary bottleneck in long-horizon GUI interaction, suggesting the dataset complexity may outpace current VLM grounding capability.
  • The exploration stage relies on a VLM for element semantic description and action selection, introducing dependency on the quality and cost of that oracle model.
  • Screen identity matching via MobileNet embeddings + Hungarian matching may misidentify dynamically loading or personalized screens as novel states, or collapse genuinely distinct states.
  • Dataset covers popular daily apps (shopping, social, video, travel, etc.) but coverage statistics are truncated in the provided text; generalization to enterprise or accessibility-heavy apps is undemonstrated.
  • The paper is an arXiv preprint; no ablation on the exploration scoring components (\(s_\text{sem}\), \(s_\text{lay}\), \(p_\text{hist}\)) is visible in the provided text.

Relevance to Vision-Language Models

SEE directly exploits VLMs as the perceptual and planning backbone for GUI agent data synthesis—using them to generate element semantic descriptions, select exploration actions, and produce multi-level language supervision from graph semantics—making it a concrete testbed for VLM grounding and spatial reasoning on real mobile interfaces. The finding that grounding accuracy (not action-type prediction) is the limiting factor on long-horizon, element-rich screens directly informs the research agenda for improving VLM spatial grounding. The graph-reuse synthesis strategy also offers a cost-efficient paradigm for generating large-scale VLM fine-tuning data without repeated screenshot re-inference, relevant to scaling VLM-based agent training. For researchers tracking VLMs, SEE provides both a challenging benchmark with 24.63 elements/screen and a reproducible data generation methodology that could be adapted to other VLM agent domains beyond mobile GUIs.