Skip to content

History-Aware Reasoning for GUI Agents

🕒 Published (v1): 2025-11-12 09:06 UTC · Source: Arxiv · Venue: AAAI 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

GUI agents trained with standard RL and Chain-of-Thought reasoning exhibit "history-agnostic" behavior—they treat each screen as an isolated perception task, ignoring prior interaction history within an episode. HAR (History-Aware Reasoning) is a training framework that forces short-term episodic memory to emerge via reflective RL over hard samples with per-instance error-correction guidelines and a model-based Memory-Augmented Reward. The resulting 3B model, HAR-GUI-3B, consistently outperforms same-size and many larger GUI agents across episodic reasoning, grounding, and understanding benchmarks.

Problem

Existing native GUI agents—even those using System-2 CoT + RL post-training—degrade multi-step episode chains to independent screen-level decisions. Their explicit CoT reasoning ignores previously executed actions (T<t), causing errors in long-horizon tasks where correct action selection depends on accumulated interaction context. RL under inference-format instructions fails to fix this because optimization pressure only adjusts action prediction, not the reasoning mode.

Method

HAR is a three-stage post-training recipe applied to a base MLLM (Qwen2.5-VL-3B-Instruct):

  1. GUI Scenario Warm-up (SFT): Injects GUI-specific knowledge via screen understanding tasks, an Action-to-Summary (Act2Sum) task where a teacher model generates goal-oriented interaction summaries, and System-2 CoT distillation from a teacher (Qwen2.5-VL-72B-Instruct).

  2. Round-1 RL (Reflection Scenario): Identifies historically incorrect samples D_his from Mwarm-up inference. For each hard sample, the teacher synthesizes up to three tailored error-correction guidelines G. A distinct reflection-format instruction (not the inference-format) presents the prior error, the guidelines, and the history to the agent. GRPO optimizes a hybrid reward: rule-based format reward Ă— (multi-scale coordinate action reward + Îł Ă— Memory-Augmented Reward). The MAR (Eq. 8) uses Qwen3-235B-A22B as a judge to verify whether the CoT explicitly references prior interaction history.

  3. Round-2 RL (Task-Mixing Training Strategy, TMTS): Switches back to inference-format instructions and mixes episodic reasoning data with screen grounding data, preventing the grounding capability regression caused by Round-1's episodic-only training.

Key Contributions

  • HAR framework: reflective learning scenario + per-instance guideline synthesis + hybrid RL reward (format + multi-scale coordinate action + model-based Memory-Augmented Reward) that transitions GUI agents from history-agnostic to history-aware reasoning.
  • HAR-GUI-3B: a 3B end-to-end native GUI model with stable short-term episodic memory, built on Qwen2.5-VL-3B.
  • Task-Mixing Training Strategy (TMTS): multi-task Round-2 RL that jointly trains episodic reasoning and grounding to prevent capability degradation.
  • Manually annotated OOD Chinese mini-program benchmark (415 tasks across 4 Alipay app categories) for out-of-distribution evaluation.
  • Ablation showing that prompt-constrained history attention (M*_GRPO) degrades performance, while autonomously shaped history-awareness (via MAR) is beneficial.

Results

  • AITW (SSR): HAR-GUI-3B 70.2 vs. MP-GUI-8B 69.2, InfiGUI-R1-3B 67.7, GUI-R1-3B 65.6, Qwen2.5-VL-3B 65.4.
  • Mind2Web (SSR, Cross-Task/Website/Domain): 42.2 / 41.2 / 44.0 vs. GUI-R1-3B 38.8 / 38.5 / 38.9 and InfiGUI-R1-3B 37.2 / 37.7 / 38.2.
  • GUI-Odyssey (SSR): 62.31 vs. InfiGUI-R1-3B 50.62, Qwen2.5-VL-7B 58.39.
  • ScreenSpot (Avg.): 83.3 vs. UI-TARS-2B 82.3, OS-Atlas-7B 82.5.
  • ScreenSpot-V2 (Avg.): 86.2 vs. UI-R1-3B 85.4, UI-TARS-2B 84.7.
  • OOD (Alipay, zero-shot SSR): HAR-GUI-3B best or second-best across all 4 categories vs. methods up to 7B; outperforms Qwen2.5-VL-72B on Repast (24.30 vs. 17.65) and matches it on Finance (21.78 vs. 21.78).
  • GUI Understanding: HAR-GUI-3B outperforms MP-GUI-8B on S2W, WS, QA, QAS; competitive or better than UI-TARS-1.5-7B on S2W and TP.

Limitations

  • Guidelines G synthesized by the teacher are unavailable at inference time; Round-2 RL is needed to bridge this train-inference gap, adding pipeline complexity.
  • The Memory-Augmented Reward requires a large LLM judge (Qwen3-235B-A22B) at training time, making RL computationally expensive.
  • History is conveyed via text summaries only (no image tokens for prior screens), which may lose visual context from earlier steps.
  • OOD benchmark is CLICK-only (no SCROLL or TYPE), limiting generality of OOD conclusions.
  • Evaluations are on existing public benchmarks; real-device online execution success rates are not reported.

Relevance to Agentic AI / LLM Agents

HAR directly addresses episodic memory—one of the core unsolved problems in building reliable long-horizon LLM agents—by showing that standard RL post-training with inference-format instructions does not fix short-term memory weakness, and that a dedicated reflection scenario with per-instance guidance synthesis and a model-based memory reward is necessary to reshape the model's reasoning mode. The Memory-Augmented Reward pattern (using a strong LLM to judge whether CoT actually leverages history) is a transferable technique applicable beyond GUI to any multi-step agentic task where episodic context should drive decisions. The TMTS finding—that mixing task types in RL prevents catastrophic forgetting of perceptual skills—is practically important for anyone building multi-capability agents with RL fine-tuning.