Skip to content

OmniGUI: Benchmarking GUI Agents in Omni-Modal Smartphone Environments

🕒 Published (v1): 2026-04-03 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

OmniGUI is the first step-level GUI agent benchmark that delivers continuous, interleaved image, audio, and video inputs at every action step on Android smartphones, rather than treating non-visual signals as pre-task references. Evaluated across 8 proprietary and open-source omni-modal models, the best achieves only 63.6% Exact Match and 33.4% episode Success Rate, exposing a hard unsolved gap in transient multimodal perception-to-action grounding.

Problem

Existing GUI benchmarks feed agents static screenshots; the few that include audio or video treat them as pre-task context (watch a tutorial, then act). Real smartphone use requires reacting to transient, synchronous signals—a specific playback moment, a spoken command, a fading percussion cue—that are tightly coupled to the exact action step and cannot be recovered from a screenshot alone. No benchmark measured this step-level omni-modal capability.

Method

The benchmark formalizes each step \(t\) as receiving observation \(S_t = (I_t, V_t, A_t, H_t)\): a high-resolution screenshot \(I_t\), a temporal video clip \(V_t\) of screen dynamics since the last action, a synchronous audio stream \(A_t\), and the prior action history \(H_t\). The agent must predict \(a_t\) from a 13-primitive action space (TAP, SWIPE_*, INPUT, HOME, BACK, TASK_COMPLETE, NONE, etc.) with continuous coordinates normalized to \([0, 1000]^2\).

Dataset construction: 10 experienced annotators ideated 709 goal episodes across 29 Android apps (bilingual: 363 Chinese, 346 English), recorded on physical devices at 30 FPS with synchronized audio. Episodes are organized along five HCI-grounded cognitive dimensions (Localization, Semantic Understanding, Cross-modal Discrimination, Temporal Reasoning, Instant Response) and receive an objective multimodal dependency label—AV-Critical (correct action unresolvable from screenshot alone), AV-Supportive (screenshot sufficient but audio/video reduces ambiguity), or AV-Present (purely static; non-visual signals are background noise)—via a modality-ablated annotation procedure (Cohen's \(\kappa = 0.84\)).

Evaluation uses a step-level teacher-forcing protocol: at each step the model receives ground-truth history \(H_t\) and predicts \(a_t\), isolating per-step multimodal perception from compounding trajectory errors. All 8 baselines (Gemini 3.0 Pro/Flash, Gemini 2.5 Pro/Flash, Qwen3-Omni, MiniCPM-o 4.5, VITA-1.5, Baichuan-Omni-1.5) use a unified prompt template with greedy decoding (temperature = 0, max 4096 tokens) and JSON-only output.

Key Contributions

  • OmniGUI benchmark: first per-step omni-modal GUI benchmark with interleaved image + audio + video at every action step.
  • 709-episode dataset: 2,579 steps across 29 apps, expert-demonstrated on physical Android devices, bilingual, with a 13-action space.
  • Objective modality dependency taxonomy: three-tier (AV-Critical / AV-Supportive / AV-Present) annotation derived from physical information availability, not model behavior; \(\kappa = 0.84\).
  • Standardized reproducible evaluation harness: unified prompt, teacher-forcing protocol, deterministic decoding; designed for plug-in of future omni-agent architectures.
  • Empirical bottleneck identification: modality ablations confirm AV-Critical degradation when non-visual inputs are removed; cross-modal interference and dual-audio processing identified as failure modes.

Results

  • Best overall: Gemini 3.0 Pro — TM 80.0%, EM 63.6%, SR 33.4%, GP 43.6%.
  • Gemini 3.0 Flash: EM 61.3%, SR 30.3% (occasionally surpasses Pro on Temporal Reasoning).
  • Gemini 2.5 Pro: EM 44.1%, SR 15.5%; Gemini 2.5 Flash: EM 37.8%, SR 12.4%.
  • Open-source leader (Qwen3-Omni): EM 32.3%, SR 5.1%; remaining open-source models SR ≤ 1.1%.
  • Dimension gap: Gemini 3.0 Pro scores EM 76.2% on Localization vs. 59.1% on Cross-modal Discrimination and 61.0% on Temporal Reasoning.
  • Modality ablation confirms taxonomy: removing audio/video causes significant EM drop on AV-Critical tasks; AV-Present tasks unaffected.

Limitations

  • Teacher-forcing only: step-level protocol does not measure autonomous rollout performance or compounding error propagation.
  • Small scale: 709 episodes / 2,579 steps is orders of magnitude smaller than AITW (715K+ steps), limiting statistical resolution on rare failure modes.
  • Android-only: no coverage of desktop or web environments.
  • No purpose-built omni-modal GUI agents: all baselines are general-purpose foundation models used as proxies; dedicated architectures do not yet exist to stress-test the benchmark.
  • GPT-4o excluded: Chat Completions API lacks native interleaved audio-visual batch ingestion; Realtime API is incompatible with deterministic step-level evaluation.

Relevance to Harnesses / Meta-Harnesses

OmniGUI directly instantiates a modular evaluation harness: a deterministic, unified inference pipeline (fixed prompt template, greedy decoding, JSON output schema, teacher-forcing) that decouples the agent-under-test from evaluation logic and enables reproducible swap-in of future architectures. The tripartite dependency taxonomy (AV-Critical / Supportive / Present) functions like a harness-level test partitioning scheme—analogous to how meta-harnesses route tasks to specialized sub-agents based on detected modality requirements. For harness designers, the cross-modal interference and dual-audio bottlenecks identified here are concrete failure modes that any omni-modal orchestration layer must handle, and the standardized step-level protocol provides a reusable scaffold for measuring whether a harness's modality-routing strategy actually improves agent fidelity.