Skip to content

Empowering GUI Agents via Autonomous Experience Exploration and Hindsight Experience Utilization for Task Planning

🕒 Published (v1): 2026-06-25 17:44 UTC · Source: Arxiv · Venue: ACL 2026 · link

Why this paper was selected

ACL 2026; GUI agents with autonomous hindsight experience for planning in open-source MLLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

PEEU (Planning Experience Exploration and Utilization) trains small open-source MLLMs (3B/7B) for GUI web navigation by having an agent autonomously explore websites, then using hindsight to retroactively align tasks to the actual collected trajectories rather than forcing trajectories to match pre-specified goals. A 7B model trained with 2k trajectories reaches 30.6% on WebVoyager, surpassing Qwen2.5-VL-32B (22.7%).

Problem

Small open-source MLLMs used as GUI web agents suffer from weak task planning and poor cross-website (OOD) generalization. Prior training approaches either use atomic-level tasks (click/type/scroll) that fail to generalize to compositional high-level tasks, or use coarse high-level tasks whose trajectories are misaligned (e.g., task requires ≥4.5 stars but trajectory only reached 4) and under-constrained because the real environment constraints are only discoverable through actual exploration.

Method

PEEU operates in two stages:

Stage 1 — Planning Tree Exploration: Given only a URL, an exploration agent (GPT-4o) reads the homepage \(s_0\) and auto-generates a task list \(D = M(s_0, \text{URL})\). It then executes goal-driven exploration, building a directed tree \(R=(V,E)\) of states and actions, yielding trajectories \(\{\tau_i\}_{i=1}^n\).

Stage 2 — Planning Experience Utilization: For each consecutive state pair \((s_t, a_t, s_{t+1})\), atomic experiences \(\epsilon_t = M(s_t, a_t, s_{t+1})\) are extracted. A trajectory-level experience \(\mu = (\epsilon_1, \ldots, \epsilon_T)\) is aggregated. Critically, instead of matching trajectories to pre-defined tasks, a mapping \(\Phi\) uses the experiences to retrospectively synthesize a new, strictly-aligned high-level task \(\tilde{d}\) ("hindsight" alignment): \(\tilde{D} = \Phi(\mu_1, \ldots, \mu_n, M)\). This ensures task–trajectory alignment and incorporates real environmental constraints (star ratings, ingredient lists, etc.) discovered during exploration. The resulting \((\tilde{d}, \tau)\) pairs fine-tune small MLLMs via SFT or GRPO.

The companion TDHAF framework defines three task granularities (low/mid/high) × two generalization axes (ID/OOD) to systematically measure compositional generalization across bottom-up, top-down, and cross-website dimensions.

Key Contributions

  • PEEU: autonomous two-stage pipeline (exploration tree → hindsight experience → aligned training data) requiring only a URL as input, no manually labeled data.
  • Hindsight task synthesis: inverting the task-trajectory relationship to guarantee alignment and strict environmental constraints, avoiding the misalignment problem of coarse methods.
  • TDHAF: a hierarchical analysis framework with three granularity levels and three generalization dimensions (ID bottom-up, ID top-down, OOD multi-level) to rigorously study compositional generalization in web navigation.
  • Empirical finding that high-level task training is necessary and sufficient for OOD generalization; low-level atomic training does not generalize upward in-domain or cross-domain.

Results

  • PEEU-SFT (7B, 2k traj): 30.6% overall on WebVoyager — beats Coarse-SFT (19.0%), Atomic-SFT (21.7%), and Qwen2.5-VL-32B Instruct (22.7%).
  • PEEU-GRPO (7B, 0.1k traj): 19.9% vs. base model 7.8% (+12.1 pp), best among all 0.1k methods.
  • PEEU-GRPO (3B, 0.1k traj): 11.1% vs. base 0.2%, outperforming Coarse-GRPO (5.6%) and Atomic-GRPO (5.5%).
  • PEEU-SFT (3B, 2k traj): 19.8% vs. Atomic-SFT 16.7% and Coarse-SFT 13.2%.
  • Prompt-only retrieval methods (Atomic-Prompt, Trajectory-Prompt) score ≤3.7% for 7B and 0.0% for 3B, worse than the untuned base — confirming retrieval augmentation alone is ineffective for small models.
  • TDHAF: training on high-level tasks yields the highest OOD multi-level generalization (7B High: Test-OOD-High Step SR = 43.2% vs. 7B Low: 33.8%). Bottom-up generalization fails: 7B Low achieves only 18.8% on Test-ID-High despite 89.6% on Test-ID-Low.

Limitations

  • Exploration phase uses GPT-4o, introducing dependence on a commercial model and cost; small MLLMs do not self-explore.
  • Maximum exploration depth of 15 steps may miss deep website states.
  • Evaluated only on WebVoyager (English, desktop browser); generalization to mobile GUIs or non-English environments is unverified.
  • TDHAF dataset is constructed for a limited set of websites; cross-domain coverage of the analysis framework is narrow.
  • GRPO experiments at 2k scale are not reported for all model sizes, leaving the GRPO ceiling unclear.

Relevance to Agentic AI / LLM Agents

PEEU directly addresses one of the central challenges in agentic systems: how to generate high-quality, grounded training data for agents acting in partially observable, interactive environments without manual annotation. The hindsight relabeling idea — collecting whatever trajectory occurs and then synthesizing a task description that is perfectly consistent with it — is analogous to Hindsight Experience Replay (HER) from RL and offers a principled bridge between exploratory behavior and supervised fine-tuning. The TDHAF analysis provides the field with a concrete measurement apparatus for compositional generalization, a property essential for any agent that must decompose novel high-level goals into executable sub-steps. The finding that atomic-skill mastery does not transfer to compositional planning challenges a common implicit assumption in tool-use and function-calling research.