Plover: Steering GUI Agents through Plan-Centric Interaction¶

🕒 Published (v1): 2026-07-16 16:48 UTC · Source: Arxiv · link

Why this paper was selected

Plan-centric GUI agent handles dynamic layouts; core capability for building robust GUI agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Plover is a mixed-initiative GUI automation system that externalizes task plans as persistent, editable artifacts rather than hiding them as internal agent state. A planner–executor architecture supports localized repair through plan editing, natural-language guidance, and screenshot annotations, preserving completed progress during correction. In benchmark failure-case repair experiments, 23 of 26 previously failed tasks improved under collaborative execution, with 17 becoming complete successes.

Problem¶

Autonomy-first vision-based GUI agents hide planning and replanning as internal model state, creating five failure modes: opaque planning (L1), silent adaptation (L2), weak spatial grounding for correction (L3), all-or-nothing correction that discards prior progress (L4), and weak verification of completion (L5). Users in long-horizon, high-friction workflows cannot detect drift, cannot apply localized corrections, and lose completed work when they re-issue prompts.

Method¶

Plover separates planning from execution through three coordinated components: an Agentic Interface, a Planner Service, and an Executor Service running on OS-level virtual machines. The central mechanism is a versioned Structured Plan artifact maintained as persistent shared state. Correction and recovery occur through Intelligent Replanning (IR) in two modes:

User-Driven IR: triggered by user intervention via (a) direct plan edits, (b) natural-language guidance, or (c) multimodal screenshot annotation; revisions target only pending steps while freezing executed history.
System-Driven IR: the executor detects non-progress autonomously and proposes a localized recovery step surfaced explicitly in the UI for user approval.

All replanning events are versioned and presented as diff-comparable proposals (current vs. proposed plan) before the user commits. The interface uses progressive disclosure—high-level activity summaries by default, detailed reasoning traces on demand—and a provenance component (Run Timeline + branching plan visualization) to maintain mental-model continuity.

A formative study with six participants (30–60 min sessions, think-aloud, screen recordings) informed the five design goals: interpretable plans (DG1), execution awareness (DG2), grounded local repair (DG3), failure-aware recovery (DG4), and continuity across repair (DG5).

Key Contributions¶

A plan-centric interaction design that treats task plans as persistent, inspectable, revisable artifacts enabling explicit supervision and localized repair during live GUI execution.
The Plover system instantiating this design with editable plan artifacts, Intelligent Replanning (user- and system-driven), and multimodal interventions (text + screenshot annotation).
An empirical characterization of recoverability across benchmark GUI-agent failure cases and scenario-based workflows, identifying which failure modes (spatial ambiguity, execution drift, misinterpreted intermediate states) are amenable to plan-centric repair vs. natural-language guidance, plan editing, or multimodal annotation.

Results¶

38 challenging benchmark tasks run under autonomous execution; 26 produced non-successful outcomes.
Re-running those 26 in collaborative (Plover-assisted) mode: 23 improved, of which 17 became complete successes and 6 became partial successes.
Average interventions per task: 2.04.
Recurring failure modes identified: spatial ambiguity in dense interfaces, execution drift in multi-step workflows, and misinterpreted intermediate states.
Scenario-based stability analysis used visual similarity and plan alignment metrics to characterize replay stability and structural plan mismatch (specific quantitative values not reported in the provided excerpt).

Limitations¶

Evaluation relies on expert-guided repair rather than a controlled user study measuring whether non-expert users can effectively apply the intervention mechanisms.
The formative study involved only six participants from a university community with prior agentic-system experience, limiting generalizability.
The system is designed for high-friction, long-horizon tasks; the paper explicitly acknowledges it is unnecessary overhead for short or low-risk workflows.
Benchmark failure-case analysis does not isolate the contribution of individual intervention modalities (edit vs. NL guidance vs. annotation) to success rates.
The paper text is truncated, so quantitative results from the scenario-based stability analysis are not fully available.

Relevance to Agentic AI / LLM Agents¶

Plover directly addresses the controllability and transparency gap in LLM-based GUI agents, which is increasingly recognized as a bottleneck for real-world deployment of agentic systems. By treating the plan as a first-class, mutable shared state rather than ephemeral reasoning, it operationalizes the "human-in-the-loop" paradigm at a granularity (step-level, screenshot-grounded) that high-level conversational correction cannot achieve. The Intelligent Replanning mechanism—particularly System-Driven IR via non-progress detection—is a concrete architectural pattern for integrating agentic self-monitoring with human oversight without reverting to full restart. This work advances the broader mixed-initiative and human-agent collaboration thread in agentic AI alongside contemporaries like Magentic-UI and CowPilot.