Skip to content

SEE: Structure-aware Exploring \& Exploiting for Long-horizon GUI Agent Trajectory Synthesis

🕒 Published (v1): 2026-07-20 15:16 UTC · Source: Arxiv · link

Why this paper was selected

Long-horizon GUI agent trajectory synthesis; structure-aware explore/exploit for scarce data

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SEE is a two-stage data synthesis framework for GUI agents that first builds an explicit screen–element transition graph via structured exploration, then synthesizes long-horizon trajectories by planning over that graph. It produces a dataset with an average trajectory length of 14.8 steps (vs. 5.5–12.8 in prior datasets) and 24.63 interactive elements per screen, substantially outperforming baselines when used for fine-tuning.

Problem

GUI agent training data is dominated by short "happy-path" trajectories from low-complexity apps, leaving the long-horizon, element-rich regime largely uncovered. Human demonstration collection is expensive and hard to scale; existing automated synthesis pipelines re-plan step-by-step on raw screenshots, causing over-sampling of common flows, spurious cycles, and prohibitive token cost for long-horizon composition.

Method

SEE operates in two decoupled stages:

Stage 1 — Structure-aware Exploration: The agent interacts with a real mobile app to construct a bipartite transition graph \(G = (V, E)\) where \(V\) contains screen nodes \(u \in \mathcal{U}\) and element nodes \(e \in \mathcal{M}\), and \(E\) contains transition edges \((u, a, u')\) and containment edges \((u, e)\). Element selection at each step is scored as \(s(e_{t,i}) = s_{\text{sem}}(e_{t,i}) + s_{\text{lay}}(e_{t,i}) - p_{\text{hist}}(e_{t,i})\), combining semantic alignment, layout priors, and history-based penalties. Screen identity is resolved via Hungarian matching on MobileNet embeddings and bounding-box positions across predefined regions of interest. After each action, a reflection step verifies the transition outcome before adding the edge to \(G\); unverified or noisy transitions are discarded.

Stage 2 — Graph-based Trajectory Synthesis: Given the completed graph, SEE samples target screen nodes, prompts an LLM to generate an ordered subgoal list and a high-level task instruction, then finds inter-subgoal paths via BFS and concatenates them into a full action sequence. Step-level action descriptions are generated cheaply by conditioning on stored node/edge semantics rather than re-parsing screenshots. Trajectory length and difficulty are controlled by adjusting subgoal count and path-length constraints.

Key Contributions

  • Structure-aware exploration algorithm that builds a verified screen–element transition graph from real mobile apps without manual trajectory annotation.
  • Graph-based synthesis pipeline that decouples structure acquisition from trajectory construction, enabling controllable long-horizon episode generation with multi-level supervision (high-level instruction, ordered subgoals, step-aligned action descriptions).
  • SEE dataset: 47K training / 5K test trajectories with average length 14.8 steps, 24.63 interactive elements per screen, and app-disjoint train/test splits — occupying the high-complexity, long-horizon quadrant unoccupied by prior datasets.
  • Reflection-based graph refinement that filters spurious edges and records interruption-handling paths (permission dialogs, overlays) as first-class transitions.

Results

  • SEE-Test benchmark: Qwen3-VL-4B fine-tuned on SEE-Train achieves 77.29% Success Rate (SR), up from 62.61% for the base model — a +14.68 pp improvement.
  • Grounding accuracy: 69.91% (FT) vs. 60.79% (base); Action Type accuracy: 97.87% (FT) vs. 93.04% (base).
  • Per-action gains from fine-tuning are especially large for Swipe (73.07% → 97.20%), Back (34.05% → 97.76%), and Tap (37.21% → 67.16%).
  • Baseline comparisons on SEE-Test: OS-Atlas-7B achieves 53.32% SR; UI-Genie 45.77%; GUI-Owl-7B 45.94% — the SEE-fine-tuned model outperforms all by a substantial margin.
  • Retrieved context (without fine-tuning) adds only +1.78% SR; combined with fine-tuning, the gain shrinks to +0.67%, indicating the fine-tuned model already internalizes structural knowledge.
  • SEE reaches 24.63 average elements per screen vs. 13.59–17.26 across six comparison datasets (Table 3).

Limitations

  • Dataset is restricted to Android mobile apps; generalization to web or desktop GUI environments is not evaluated.
  • The exploration stage requires live interaction with real apps, so coverage depends on how thoroughly the exploration agent discovers app states — rare or deeply nested screens may still be under-represented.
  • Screen identity resolution relies on MobileNet embeddings from stable UI regions; highly dynamic or visually inconsistent apps could confuse the matching, producing duplicate or merged nodes.
  • The paper text is truncated before full ablations and coverage statistics are reported (Figure 4 is cut off), so the breadth of app categories covered is not fully visible.
  • All fine-tuning results use a single backbone (Qwen3-VL-4B); cross-architecture generalization is not demonstrated.

Relevance to Agentic AI / LLM Agents

SEE addresses a core bottleneck for training capable GUI agents: the scarcity of long-horizon, high-complexity trajectory data, which constrains agents to shallow task policies. The framework's explicit decoupling of world-model construction (graph exploration) from policy training data generation (graph-based synthesis) is a principled step toward scalable agentic data pipelines that do not require expensive online rollouts or human demonstrations at synthesis time. The reflection-based graph refinement and interruption-handling transitions are especially relevant for robust real-world deployment, where transient overlays and prerequisite-dependent flows are common failure points for current LLM agents.