Skip to content

GUI Exploration Lab: Enhancing Screen Navigation in Agents via Multi-Turn Reinforcement Learning

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

GUI Exploration Lab (GE-Lab) is a simulation environment engine for GUI screen-navigation research that models app screens as tree-structured graphs with fully observable navigation metadata. The paper establishes a three-stage training paradigm—SFT → single-turn RL (ST-RL) → multi-turn RL (MT-RL)—and demonstrates that MT-RL, trained via online interactive rollouts with sparse goal rewards, yields the best generalization to OOD environments and real-world interactive benchmarks.

Problem

Real-world GUI platforms (mobile apps, PC software) are proprietary and structurally opaque, making it prohibitively costly to obtain complete navigation graphs and high-quality trajectory data for training GUI agents. This prevents systematic ablation of training strategies for screen-navigation tasks, which require multi-step planning far beyond single-screen grounding.

Method

GE-Lab renders tree-structured environments where nodes = screens and edges = clickable icon transitions. Icon images, names, and positions are randomized from an uncommon internet-sourced library to prevent semantic shortcutting. The engine provides full environment metadata (navigation graph, reward signal) for three training regimes:

  1. SFT: Expert trajectories (shortest and redundant paths) are synthesized from the graph and used to fine-tune Qwen2.5-VL-7B-Instruct, memorizing edge-level transition knowledge.
  2. ST-RL (GRPO-based): Pre-constructed trajectories are used; the model rolls out only the final action. A composite 4-component reward evaluates action type, click coordinate accuracy, icon intent match, and format correctness.
  3. MT-RL: Online rollouts over up to 12 interaction rounds; the model generates the full observation-action sequence interactively. Reward is sparse (+1 for reaching target screen, 0 otherwise). Curriculum learning and data proportion adjustment mitigate reward hacking (e.g., action-space restriction to prevent trivial exploit loops).

Agent protocol uses ReAct-style chain-of-thought (explanation + action command), with history stored as text to keep context under ~1k tokens per step.

Key Contributions

  • GE-Lab engine: Flexible, fully observable simulation with customizable graph topology, randomized icon semantics, and accurate reward computation—first GUI simulation engine supporting MT-RL interactive training.
  • Empirical training paradigm: SFT memorizes, ST-RL generalizes, MT-RL explores—validated across ID, OOD, and interactive benchmarks.
  • OOD robustness analysis: Five OOD variants (Env-Image, Env-Name, Env-Position, Env-Noise, Env-Base) quantify sensitivity to each perturbation type.
  • Real-world transfer: Zero-shot evaluation on 1,569 real GUI samples and continual training on 24k real-world samples with MT-RL achieving best average score.
  • SFT–RL interaction finding: Moderate SFT (fewer epochs) yields better RL initialization than over-trained SFT, due to preserved policy plasticity.

Results

  • In-distribution (ID): SFT 98.89%, ST-RL 97.63%, MT-RL 67.33% (Overall); MT-RL deliberately underfits ID to avoid overfitting.
  • OOD (Env-Base): MT-RL 63.25% vs. ST-RL 63.06% vs. SFT 55.45% (Overall); ~14% relative gain over SFT for RL methods.
  • OOD with icon/name change: All methods drop to ~32–34% overall (Path drops to ~7%), showing icon identity is the critical knowledge dependency.
  • Interactive benchmark (Pass@1 / Pass@5): MT-RL 17.47/25.16 > ST-RL 17.22/22.34 > SFT 14.30/20.86.
  • Long paths (7 steps, Pass@5): MT-RL 8.33% vs. ST-RL 3.39% vs. SFT 1.30%—MT-RL advantage grows with path length.
  • Real-world continual training average: MT-RL-Continue-Train 72.03% > ST-RL 70.92% > SFT 70.87%; AndroidWorld: MT-RL 15.51% vs. ST-RL 14.65% vs. SFT 12.93%.
  • Non-fine-tuned baselines (GPT-4o, Claude-3.7-Sonnet, Gemini-2.0-flash-thinking) score near random on simulated navigation tasks.

Limitations

  • GE-Lab uses synthetic, randomly named icons that are intentionally uncommon—the simulated visual domain diverges significantly from real GUI aesthetics, limiting direct icon-level transfer.
  • MT-RL underperforms on ID benchmarks, indicating the sparse reward signal sacrifices memorization of specific known paths.
  • Icon/name OOD perturbations cause catastrophic performance drops (~7% Path success) for all methods, revealing that fundamental icon-identity knowledge is brittle.
  • Real-world interactive evaluation (AndroidWorld) scores remain low (≤15.51%), suggesting the sim-to-real gap is only partially closed.
  • Navigation graphs are restricted to tree structures; cyclic real-world navigation topologies are not modeled.

Relevance to Vision-Language Models

This paper directly addresses how to train LVLMs for agentic GUI control, advancing the shift from static grounding tasks to sequential multi-screen navigation. The finding that MT-RL with sparse rewards outperforms trajectory-imitation SFT on long-horizon tasks has broad implications for VLM agent training pipelines beyond GUI—any multi-step, partially observable task where environment interaction is feasible. The SFT→ST-RL→MT-RL paradigm and the empirical finding that moderate (not maximal) SFT preserves RL plasticity is a practical design principle for VLM post-training. The simulation engine addresses the chronic bottleneck of GUI-specific training data scarcity, a recurring obstacle in LVLM-based agent research.