Skip to content

Training High-Level Schedulers with Execution-Feedback Reinforcement Learning for Long-Horizon GUI Automation

🕒 Published (v1): 2025-11-27 09:01 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CES (Coordinator-Executor-State Tracker) is a multi-agent framework that decouples long-horizon GUI automation into three specialized roles — strategic planner, action executor, and state tracker — trained via a staged execution-feedback RL algorithm. Rather than training a single overloaded policy, it freezes a pretrained Executor and trains only the high-level Coordinator and State Tracker using reward signals derived from actual execution outcomes. The resulting high-level components are plug-and-play and improve long-horizon task success across multiple Executor models.

Problem

Single-agent GUI models suffer from two compounding failures on long-horizon tasks: (1) responsibility coupling and capability conflict — optimizing one unified network for both high-level planning and low-level pixel-space grounding creates conflicting gradients and catastrophic capability collapse; and (2) lack of task state awareness — relying on raw historical action sequences or screenshots is insufficient because screenshots are semantically ambiguous (e.g., a Home screen repeats), making it impossible for the agent to determine its position within a long task trajectory.

Method

CES structures GUI automation as an OS-inspired loop: the Coordinator (Qwen2.5-VL-7B) translates high-level user intent into atomic instructions \(l^t = \pi_c(q, m^{t-1}, s^t)\); the Executor (any frozen pretrained GUI model) executes those instructions \(u^t = \pi_e(l^t, s^t)\); the State Tracker (Qwen3-4B) compresses execution outcomes into a natural-language state summary \(m^t = \pi_s(q, m^{t-1}, u^t)\) that replaces raw screenshot history.

Training uses staged execution-feedback RL (GRPO-based): - Stage 1: Freeze the Executor and State Tracker (supplying ground-truth states); train the Coordinator using an Execution-Feedback Reward \(R = \alpha_1 R_{\text{format}} + \alpha_2 R_{\text{executor}}\), where \(R_{\text{executor}} = \gamma_1 R_{\text{type}} + \gamma_2 R_{\text{param}}\) is derived from the Executor's action correctness. - Stage 2: Freeze the Coordinator; train the State Tracker by back-propagating the same Execution-Feedback Reward through the fixed Coordinator–Executor chain, teaching the State Tracker to produce summaries maximally useful to the Coordinator's fixed policy.

Both stages begin with a warm-up SFT phase (1 epoch, lr=5e-5) before RL (10/5 epochs, lr=1e-6), run on 8×80G GPUs.

Key Contributions

  • CES multi-agent framework: OS-inspired architecture separating CPU (Coordinator), I/O (Executor), and Dynamic Memory (State Tracker); the high-level components are plug-and-play with any Executor.
  • State Tracker: A dedicated language model performing dynamic context compression into high-semantic natural-language state summaries, eliminating reliance on raw screenshot sequences.
  • Staged execution-feedback RL: Decoupled two-stage GRPO training using verifiable execution rewards to optimize abstract high-level agents without direct evaluation of intermediate outputs.
  • Empirical demonstration that screenshots alone are insufficient state representations: temporal-ordering accuracy drops dramatically as step interval increases.

Results

Against benchmarks AITZ, AMEX, and GUI-Odyssey (Goal Rate / Success Rate):

  • CES + GUI-R1-7B Executor vs. GUI-R1-7B baseline:
  • AITZ: GR 64.44 vs. 52.73 (+11.71), SR 64.58 vs. 54.92 (+9.66)
  • AMEX: GR 77.57 vs. 67.26 (+10.31), SR 61.64 vs. 57.12 (+4.52)
  • GUI-Odyssey: GR 79.24 vs. 65.49 (+13.75), SR 63.82 vs. 43.64 (+20.18)
  • CES + GUI-Owl-7B: +14.16/+12.81 GR/SR on AITZ, +14.27/+15.43 on GUI-Odyssey vs. baseline.
  • CES outperforms GPT-4o, OS-Atlas-7B, UI-R1-3B, GUI-Owl-7B, and SWIRL on GUI-Odyssey SR (63.82 vs. SWIRL's 51.65 and GUI-Owl-7B's 45.96).
  • CES-P ablation (prompt engineering only, no RL training): degrades or offers marginal gains, confirming that RL-optimized specialization is critical; e.g., UI-R1-3B CES-P drops GR by −17.71 on AMEX vs. CES's +10.16 gain.
  • CES + GPT-4o (replacing GUI-R1-7B Executor with GPT-4o): GUI-Odyssey GR 70.37 → 79.24 with CES vs. 70.37 standalone.

Limitations

  • The Executor is entirely frozen during training; the reward signal is indirect and depends on the Executor's own reliability — a weak Executor generates noisy rewards.
  • Stage 2 trains the State Tracker conditioned on a fixed Coordinator policy; if the Coordinator is later swapped, the State Tracker may produce suboptimal summaries for the new planner.
  • Warm-up SFT for both agents requires annotated trajectory data with ground-truth state labels, partially retaining the annotation burden they aim to eliminate.
  • Evaluation is limited to three mobile/GUI benchmarks; generalization to desktop or web environments is not assessed.
  • The two-stage sequential training cannot jointly optimize Coordinator and State Tracker interactions.

Relevance to Harnesses / Meta-Harnesses

CES is a concrete instance of a meta-harness pattern: it wraps any existing low-level executor (the "tool") with trained high-level orchestration components (Coordinator + State Tracker) that handle scheduling, decomposition, and state management — exactly the role a meta-harness plays over sub-agents or sub-tools. The staged execution-feedback RL approach is directly applicable to harness design: it shows how a harness's control layer can be trained using verifiable downstream signals without touching the wrapped tool. The plug-and-play property — demonstrated by swapping GUI-R1-7B, GUI-Owl-7B, and GUI-Owl-32B as Executors — is the defining structural property of a composable meta-harness. The State Tracker's role as a persistent, compressed execution log addresses a recurring challenge in harness design: maintaining coherent state across long multi-step pipelines without context overflow.