Skip to content

EchoTrail-GUI: Building Actionable Memory for GUI Agents via Critic-Guided Self-Exploration

🕒 Published (v1): 2025-12-22 13:42 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

EchoTrail-GUI is a three-stage framework that gives GUI agents persistent, actionable memory by having them autonomously explore Android environments, filtering collected trajectories with a critic model, and injecting the top-K retrieved trajectories as in-context guidance at inference time. This "learn–remember–apply" loop is fully automated and training-free, converting stateless VLM-based agents into experience-aware systems. On AndroidWorld, GPT-4o augmented with EchoTrail-GUI reaches 51.7% SR, the highest reported among training-free agents.

Problem

Contemporary GUI agents suffer from "digital amnesia": each task is treated in isolation with no mechanism to accumulate or reuse successful operational experience. Two sub-problems compound this: (1) high-quality trajectory data is hard to collect at scale without manual annotation, and (2) even when trajectory corpora exist, agents cannot dynamically retrieve and apply them, relying instead on static handcrafted prompts.

Method

The framework operates in three sequential stages:

  1. Critic-Guided Self-Exploration: An exploration agent (Gemini 2.5 Flash) generates trajectories on Android environments using a "Progressive Intent Focus" — starting in curiosity-driven mode then switching to target-focused mode after a warmup. A dual-database system (volatile Dproc for in-progress trajectories; permanent Dmem for archived successes) provides real-time guidance to the explorer. Each completed trajectory is scored by a critic/reward model (Gemini 2.5 Flash Lite, 1–5 scale); only trajectories with score ≥ 4 enter Dmem. Stored trajectories are abstracted into (screen description, intent summary, action) tuples (~1k tokens/trajectory vs. >10k for raw screenshots).

  2. Dynamic Memory Injection: Given a new task instruction I, a hybrid dense+sparse retrieval system (FAISS cosine similarity over Qwen3-Embedding-4B embeddings + BM25) scores each trajectory in Dmem and retrieves the top-K=2 most relevant ones. Retrieved trajectories are formatted into step-by-step {interface description, intent, action} guides.

  3. Memory-Augmented Inference: The retrieved memories are injected into the backbone agent's prompt alongside the current screenshot (described via Qwen3-30B summarizer), action history, and task instruction. Any off-the-shelf VLM can serve as the backbone; the paper uses GPT-4o and Qwen2.5-VL-72B-Instruct.

The EchoTrail-4K dataset was built from 4,143 exploration episodes (avg. 4.8 steps, ~4.3 min/valid trajectory per device).

Key Contributions

  • EchoTrail-GUI: a fully automated, training-free, plug-and-play memory framework for GUI agents with no human supervision required during knowledge base construction.
  • EchoTrail-4K: a curated dataset of 4,000+ high-quality abstracted Android trajectories.
  • Critic reliability analysis: Cohen's κ ≈ 0.72 vs. human experts; open-source Qwen3-VL-235B-A22B achieves F1=0.86/κ=0.68, matching GPT-4o as a critic.
  • Demonstration that injecting unfiltered low-quality memories is worse than no memory at all (ablation).

Results

AndroidWorld (Success Rate): - EchoTrail-GUI + GPT-4o: 51.7% SR — highest among all training-free agents; outperforms GUI-explorer (47.4%), URST (46.6%), AndroidGen (46.8%), M3A (40.5%) - EchoTrail-GUI + Qwen2.5-VL-72B: 46.6% SR (training-free), matching fine-tuned UI-TARS-72B-SFT (46.6%)

AndroidLab (SR / Sub-SR / RRR / ROR): - EchoTrail-GUI + GPT-4o: 48.1% SR, 50.7% Sub-SR, 97.9% RRR, 88.5% ROR — surpasses AutoGLM (36.2%) and GPT-4o baseline (31.2%) - EchoTrail-GUI + Qwen2.5-VL-72B: 37.5% SR, 41.1% Sub-SR — more than doubles the backbone's 23.9% SR

Ablation (AndroidWorld, Qwen2.5-VL-72B): - Full system: 46.6%; w/o critic filtering: 31.0% (below no-memory baseline of 34.1%); w/o hybrid retrieval: 40.5%; w/o real-time guidance: 42.7%

Memory K sensitivity: K=2 optimal; K=0 baseline 34.1%, K=2 peaks at 46.6%, performance degrades beyond K=3.

Limitations

  • Memory built from a single device running sequentially (~4.3 min/trajectory); scaling to diverse device configurations or app versions not addressed.
  • Exploration agent (Gemini 2.5 Flash) and critic (Gemini 2.5 Flash Lite) are called at construction time — inference is training-free but memory building still requires substantial LLM API calls.
  • Performance gains are smaller on hard-difficulty tasks (15.8% for full EchoTrail-GUI vs. 13.2% backbone on AndroidWorld hard split), suggesting memory alone cannot overcome fundamental reasoning gaps on the hardest tasks.
  • Memory retrieval relies on semantic similarity of task instructions; tasks with novel or ambiguous phrasings may retrieve poor matches.
  • The framework is evaluated only on Android environments (AndroidWorld, AndroidLab); generalization to desktop or web GUIs is claimed but not demonstrated.
  • K=2 is tuned on the same benchmark used for evaluation; optimal K may vary across domains.

Relevance to Agentic AI / LLM Agents

EchoTrail-GUI directly addresses one of the central open problems in agentic AI: how to give agents persistent, reusable operational memory without expensive fine-tuning or human annotation. The critic-guided self-exploration loop is a concrete implementation of autonomous self-improvement — an agent bootstrapping its own training data through environmental interaction with quality control. The non-parametric RAG-over-trajectories approach is model-agnostic and complementary to model scaling, showing that structured memory retrieval can match or exceed fine-tuned specialist agents. This work is closely related to emerging lines on agent self-play, offline trajectory datasets, and tool-augmented in-context learning, and the core principle — critic-filtered autonomous exploration yields reusable experience — is broadly applicable beyond GUI agents to any interactive LLM agent facing statelessness.