Training High-Level Schedulers with Execution-Feedback Reinforcement Learning for Long-Horizon GUI Automation

🕒 Published (v1): 2025-11-27 09:01 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper proposes CES (Coordinator-Executor-State Tracker), a multi-agent framework for long-horizon GUI automation that decouples high-level scheduling from low-level execution. The key innovation is a staged execution-feedback RL algorithm that trains only the Coordinator (planning) and State Tracker (context compression) while keeping the Executor frozen, using verifiable execution outcomes as reward signals. CES functions as a plug-and-play module compatible with arbitrary Executor models.

Problem

Single-agent GUI models suffer from two compounding failures on long-horizon tasks: (1) responsibility coupling — a unified policy must simultaneously master high-level task planning, multi-step reasoning, GUI grounding, and precise action execution, creating conflicting optimization objectives that cause capability collapse; (2) state unawareness — agents relying on raw screenshots or low-level action histories (e.g., Click(x,y)) cannot reliably infer task progress, especially when the same ambiguous screens (Home screen, OOD interfaces) recur. A preliminary temporal reasoning experiment confirms this: accuracy of ordering two screenshots from the same trajectory drops dramatically as the step interval grows.

Method

CES decomposes GUI automation into an MDP solved by three specialized agents in a closed loop:

• Coordinator (Qwen2.5-VL-7B): receives the user instruction q, the previous state summary m^{t-1}, and the current screenshot s^t, and outputs a clear atomic instruction l^t for the Executor.
• Executor (frozen, e.g., GUI-R1-7B): translates l^t into a concrete GUI action a^t; treated as a black-box tool.
• State Tracker (Qwen3-4B, language model only): observes the Executor output u^t and updates a natural-language task state summary m^t, compressing visual/action history into high-semantic memory.

Training follows a staged execution-feedback RL strategy using GRPO: 1. Warm-up SFT: both Coordinator and State Tracker learn their output format from annotated trajectories. 2. Stage 1 — Coordinator RL: frozen Executor produces action-level rewards (R_type + R_param); ground-truth state annotations provide m^{t-1}; GRPO optimizes the Coordinator's planning policy. 3. Stage 2 — State Tracker RL: frozen Coordinator and Executor form a chain; the same execution-feedback reward back-propagates through the chain to optimize the State Tracker, teaching it to produce summaries maximally useful to the trained Coordinator.

The reward function is R = α₁·R_format + α₂·R_executor with α₁=0.1, α₂=0.9.

Key Contributions

• CES multi-agent framework with OS-inspired role separation (CPU/I/O Device/Dynamic Memory analogy).
• A dedicated State Tracker that maintains high-semantic natural-language state summaries, eliminating reliance on redundant screenshot histories.
• Staged execution-feedback RL: trains high-level agents using verifiable downstream execution outcomes as reward, without labeling intermediate planning quality.
• Plug-and-play design: trained Coordinator + State Tracker can wrap any frozen Executor model and improve its long-horizon performance.

Results

vs. GUI-R1-7B (base Executor): - AITZ: Type 52.73→64.44 (+11.71%), SR 54.92→64.58 (+9.66%) - AMEX: GR 67.26→77.57 (+10.31%), SR 43.69→48.48 (+4.79%) - GUI-Odyssey: GR 65.49→79.24 (+13.75%), SR 43.64→63.82 (+20.18%)

vs. using GPT-5 as Coordinator/State Tracker via prompting (same GUI-R1-7B Executor): - CES outperforms GPT-5 prompting on all metrics; GPT-5 prompting yields only ~4% average Type gain with instability.

Generalization across Executors (Table 2): - UI-R1-3B Executor: SR on GUI-Odyssey +5.55% (vs. baseline), despite CES-P degrading by −18.05%. - GUI-Owl-7B Executor: SR on GUI-Odyssey +10.83%. - GUI-Owl-32B Executor: SR on AMEX +17.15% (via CES-P), SR on GUI-Odyssey +17.15% (via CES).

Ablation (Table 3, AMEX): - w/o Coordinator: SR drops 48.48→33.27 (−15.21%) - w/o State Tracker: SR drops 48.48→42.08 (−6.40%) - w/o RL (SFT only): SR drops 48.48→36.54 (−11.94%)

Failure analysis: CES reduces State Loss errors from 14%→2% and Planning Errors from 12%→4% relative to GUI-R1-7B baseline.

Limitations

• The Executor is frozen; end-to-end joint optimization of all three components is left as future work.
• Stage 1 relies on ground-truth annotated state summaries to bootstrap Coordinator training — this annotation requirement may not generalize to fully automated pipelines.
• Evaluation is confined to mobile GUI benchmarks (Android); generalization to desktop/web GUI environments is unvalidated.
• The State Tracker operates solely on text (Executor outputs), not screenshots — it cannot recover from Executor perception errors or hallucinated action descriptions.
• Performance bottleneck shifts to the Executor's perceptual limitations (Perception Error and Generalization Failure rates unchanged by CES).

Relevance to Agentic AI / LLM Agents

CES directly addresses the single-agent capability overload problem — a core challenge in building agents that must handle long-horizon, multi-step tasks — by establishing that specialized role decoupling combined with RL trained on downstream execution feedback is more effective than scaling a monolithic policy or prompting large frontier models. The staged execution-feedback RL strategy is a practical template for the broader problem of training intermediate agents in a pipeline where only the terminal outputs are verifiable, relevant to any agentic system with an unobservable reward chain. The plug-and-play framing — where the trained high-level scheduler wraps arbitrary low-level executors — aligns with the growing trend of modular agent architectures and hints at a general pattern: train orchestration logic separately from action primitives. The explicit State Tracker design is also a concrete architectural response to context accumulation failures common in long-horizon LLM agent deployments.