GUIDE: A Benchmark for Understanding and Assisting Users in Open-Ended GUI Tasks¶

🕒 Published (v1): 2026-03-26 19:37 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Why this paper was selected

CVPR 2026; GUIDE: benchmark for GUI agents assisting open-ended creative software tasks

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

GUIDE is a benchmark evaluating multimodal LLMs on three hierarchical GUI-understanding tasks: detecting user behavioral state, predicting user intent, and deciding when/how to assist. Built from 67.5 hours of novice screen recordings across 10 creative/analytical applications, it exposes a critical gap—current MLLMs peak at only 44.6% accuracy on behavior state detection—while showing that structured user context can lift help prediction by up to 50.2 percentage points.

Problem¶

Existing GUI agent benchmarks focus on automating closed-ended tasks from expert demonstrations, ignoring how real users actually work: iteratively, with shifting goals, and with a need to retain agency. No benchmark evaluates whether models can perceive a user's behavioral state, infer their underlying intent, and proactively decide when and how to assist during open-ended, exploratory workflows. This leaves the "understanding" layer of GUI agents entirely uncharacterized.

Method¶

The authors collect 120 screen recordings (67.5 hours) from 54 novice users on 40 open-ended tasks across 10 applications (Photoshop, Figma, PowerPoint, Premiere Pro, Excel, etc.), each session accompanied by think-aloud narrations. A human–AI collaborative pipeline (Gemini-2.5-Pro + two human annotators, 96.1% agreement) annotates three task layers:

Behavior State Detection: Nine-class taxonomy (4 high-level phases: Planning, Execution, Problem-Solving, Evaluation) derived via iterative human taxonomy + LLM-generated taxonomy merger, validated against Norman's Seven Stages and Bloom's cognitive hierarchy. Dataset: 1,800 balanced segments (200 per class).
Intent Prediction: MCQ (4 options) over the user's immediate sub-goal, inferred from narration-aligned segments via Gemini-2.5-Pro + human review; 1,300 validated instances.
Help Prediction: Two subtasks—binary help-need detection and MCQ help-content prediction (4 options). Annotations combine explicit signals (user switching to Google/YouTube/ChatGPT) and implicit narration cues (expressed confusion); 1,000 validated instances with 66% help-needed.

All benchmark evaluation is vision-only (32 uniformly sampled frames per segment, no audio), simulating real-world conditions. Eight MLLMs are evaluated zero-shot. Three progressive context conditions test whether providing behavioral state and intent labels as input improves downstream predictions.

Key Contributions¶

GUIDE benchmark covering three hierarchically ordered tasks (behavior → intent → help) on authentic novice GUI workflows, distinct from existing expert-demonstration automation benchmarks.
A validated 9-state behavior taxonomy grounded in cognitive theory (Norman's stages, Bloom's hierarchy) for GUI workflows.
67.5 hours of novice screen recordings with aligned think-aloud narrations across 5 application categories and 40 open-ended tasks.
Zero-shot evaluation of 8 state-of-the-art MLLMs (Gemini-2.5-Pro/Flash, GPT-4o/mini, Claude-4.5-Sonnet, Qwen3-VL-8B, InternVideo2.5-8B, InternVL3-8B), establishing baselines and revealing context-sensitivity.
Demonstration that layered user context (behavior state → intent) is a high-leverage signal for assistance decisions.

Results¶

Behavior State Detection: Best model is Claude-4.5-Sonnet at 44.61% accuracy (9-class). Adding prior-segment history yields marginal gains (e.g., 44.61% → 45.63%).
Intent Prediction: Claude-4.5-Sonnet reaches 71.39% (default); adding behavior state context improves to 72.62%.
Help Need Detection: Gemini-2.5-Pro best at 69.82% accuracy in default setting; providing behavior state lifts it to 84.73%; behavior + intent yields 82.38%. GPT-4o reaches 87.79% with behavior state.
Help Content Prediction: Claude-4.5-Sonnet best at 55.00% default; adding behavior state → 62.17%; adding behavior + intent → 82.79%.
Largest single-model gain from context: help prediction boosted by up to 50.2pp (lowest-performing model with full context vs. default).
Open-source 8B models (InternVideo2.5, InternVL3) substantially lag proprietary models on all tasks.

Limitations¶

Benchmark is constructed via automated annotation (Gemini-2.5-Pro) followed by human review; residual annotation noise from the LLM-in-the-loop pipeline is possible.
Evaluation uses static 32-frame uniform sampling from offline video segments; temporal dynamics within segments are not fully preserved.
The online (progressive) evaluation setting is mentioned but results are only partially described in the provided text.
User pool is limited to self-reported novices from Prolific and one institution, which may not generalize to diverse expertise levels or cultural contexts.
The 9-class behavior taxonomy, while grounded in theory, was partially generated by an LLM and may not exhaustively cover all behavioral states.
No fine-tuned models are evaluated; it is unclear how much performance ceiling remains from in-domain training.

Relevance to Agentic AI / LLM Agents¶

GUIDE directly probes the perceptual and reasoning capabilities that a proactive GUI agent must have before it can act helpfully: understanding what state a user is in, what they want, and whether to intervene—rather than just executing instructions. The finding that structured context (behavioral state + inferred intent) can lift help prediction accuracy by over 50 percentage points motivates a chain-of-thought or hierarchical inference architecture for LLM agents, where intermediate world-model layers (user state, intent) are maintained and surfaced explicitly before generating assistance. This connects to the broader line of work on mixed-initiative, human-in-the-loop agentic systems, and establishes a concrete evaluation protocol for the "understand before act" paradigm that the field increasingly advocates.