Skip to content

FlashAdventure: A Benchmark for GUI Agents Solving Full Story Arcs in Diverse Adventure Games

🕒 Published (v1): 2025-09-01 01:33 UTC · Source: Arxiv · Venue: EMNLP 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

FlashAdventure is a benchmark of 34 free Flash-based adventure games that evaluates GUI agents on completing full narrative story arcs from start to finish. It introduces CUA-as-a-Judge—an automated evaluation agent—and COAST, a clue-memory agentic framework, to address the "observation-behavior gap" where agents must act on information acquired much earlier in gameplay. All tested agents achieve near-zero success rates, with a large gap relative to human performance (97.1% success).

Problem

Existing game benchmarks for GUI agents cover at most 12 games, rarely require completing a full story arc, and lack automatic evaluation (relying on manual annotation). They also fail to stress-test long-term temporal dependencies—cases where an agent observes a clue at step \(t\) but must act on it at step \(t + \Delta\) where \(\Delta\) can be hundreds of steps. Prior room-escape benchmarks that do feature story completion are too short (5.7–257.8 human steps) to expose this "observation-behavior gap."

Method

Benchmark construction. 34 Flash-based adventure games (mystery/detective, hidden object, room escape, visual novel, simulation) are sourced from the FlashPoint Archive. Gameplay is formalized as a POMDP \(\langle S, A, O, \Omega, \mathcal{T}, R \rangle\) where the agent receives only raw RGB frames and issues low-level GUI actions (clicks, keystrokes). Each non-simulation game has 4–12 discrete milestones (avg 6.7); simulation games use a normalized HUD score.

CUA-as-a-Judge. After an agent episode ends at state \(s_T\), a separate judge agent (Claude-3.7-Sonnet computer-use) resumes from \(s_T\), directly interacts with the game environment, and sequentially verifies predefined milestones. The completion score is \(K_m / N_m\) where \(K_m\) is the last consecutive milestone satisfied before failure.

COAST (Clue-Oriented Agent for Sequential Tasks). A three-phase Seek-Map-Solve cycle: 1. Clue Seeker (\(\pi^\text{seek}_\theta\)): explores for \(N_\text{seek}\) steps, extracting structured clue records (name, description, location, interactability, usage hint) into a shared memory \(M\). 2. Clue Mapper (\(\pi^\text{map}_\phi\)): identifies up to \(K\) promising (clue \(\in M\), observation \(o_t\)) pairs from the trajectory, generates one subtask hypothesis per pair into candidate set \(G\). 3. Problem Solver (\(\pi^\text{solve}_\psi\)): executes each subtask \(g \in G\) for \(N_\text{solve}\) steps; resolved goals are cached in \(G_R\) to prevent repetition. The cycle repeats until the step budget \(T\) is exhausted.

Key Contributions

  • Largest video-game GUI benchmark (34 diverse games), all free-to-play and screenshot-only, requiring full story-arc completion.
  • Formal characterization of the observation-behavior gap (avg 251.1 steps between clue acquisition and reward in human play).
  • CUA-as-a-Judge: automated oracle evaluator achieving 94.00% accuracy, Spearman \(\rho = 0.9912\), Pearson \(r = 0.9999\) vs. human judges over 300 episodes.
  • COAST framework: structured long-term clue memory with a Seek-Map-Solve cycle that improves milestone completion over all baselines.
  • Human baseline: 13 participants, avg 1,142 steps, 26 min, 97.1% success—establishing a clear target.

Results

  • All evaluated agents (Claude-3.7-Sonnet CUA, OpenAI CUA, UI-TARS-1.5-7B, Cradle, Agent S2) achieve near-zero success rates across 34 games; the best agents only completed two hidden-object games (Grim Tales series).
  • Milestone completion rates for all baselines are significantly below human performance.
  • COAST improves milestone completion rates over comparable baselines by bridging the observation-behavior gap via clue memory, though exact per-agent delta numbers are in Table 2 (not fully reproduced in the provided text).
  • Human avg observation-behavior gap: 251.1 steps; individual examples show gaps of 99 and 421 steps.
  • CUA-as-a-Judge validated on 300 sampled episodes (8–9 per game): accuracy 94.00%, Spearman 0.9912, Pearson 0.9999.

Limitations

  • Maximum step budget of 1,000 may still be insufficient for the most complex games, given humans average 1,142 steps.
  • CUA-as-a-Judge shows reduced reliability on counting-based milestones requiring simultaneous assessment of multiple items.
  • COAST's clue memory is unbounded but currently averages ~12.8K tokens; the approach may not scale cleanly to games with far denser clue distributions.
  • Benchmark is Flash-game specific; Flash's standardized rendering and relatively static layouts may not reflect the full diversity of modern GUI environments.
  • Text truncation in the provided paper omits full quantitative COAST vs. baseline tables, so precise improvement margins cannot be stated here.

Relevance to Harnesses / Meta-Harnesses

CUA-as-a-Judge is a direct instance of a judge harness: a separate agent that wraps around a primary agent's episode, re-enters the environment post-execution, and programmatically verifies outcomes—an automated meta-evaluation layer analogous to test-harness oracles in software pipelines. COAST itself is an agentic harness with a structured three-phase cycle (Seek-Map-Solve) that coordinates three sub-agent roles (Clue Seeker, Clue Mapper, Problem Solver) around a shared persistent memory store, demonstrating how meta-level orchestration logic (cycling, filtering resolved goals, gating phases) can be decoupled from the backbone VLM. Together, these two components illustrate how harnesses enable both automated evaluation and structured multi-phase agent execution in long-horizon tasks—patterns directly applicable to general-purpose agent meta-harness design.