Skip to content

SMAN-Bench: A Cross-System Benchmark for Mobile Agents under Single- and Multi-path, Ambiguous, and Noisy Tasks

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Cross-system mobile agent benchmark with ambiguous and noisy real-world tasks

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SMAN-Bench is an offline benchmark for VLM-based mobile GUI agents covering 12,856 instructions across single-path, multi-path, noisy, and ambiguous task settings. It introduces a slot-based instruction generation method (GIAS) that annotates unlabeled graph-structured trajectories from Mobile3M, enabling stable step-level reward computation without real-device instability. Evaluation of 20+ models shows that even the strongest VLMs (GPT-4o, Qwen2-VL-72B) achieve low success rates, especially on complex and noisy tasks.

Problem

Existing mobile agent benchmarks suffer from two complementary failures: (1) online benchmarks offer multi-path flexibility but produce unstable step rewards due to OS/app update noise and environment drift; (2) offline benchmarks provide stable rewards but constrain evaluation to a single annotated trajectory, penalizing semantically valid alternative solutions and inflating scores for trajectory-overfitting agents. Additionally, neither category models the realistic presence of advertisement noise or the need for proactive clarification under ambiguous instructions.

Method

SMAN-Bench is built on Mobile3M, a directed graph corpus of 3M Android UI pages and 20M interactions across 49 apps. The GIAS (Generating Instructions From Action Sequences) pipeline: (1) samples trajectories from graph nodes sharing a common start; (2) annotates each GUI page with VLM-generated descriptions; (3) infers action intent between adjacent states; (4) extracts slot values (specific entity strings like "16GB+512GB", "Midnight version") from GUI diffs; (5) fills predefined instruction templates with these slots; (6) deduplicates via similarity threshold \(\text{Sim}(I_i, I_j) \geq \tau\).

Multi-path evaluation merges multiple single trajectories into a unified graph by identifying homogeneous key nodes via (a) BM25 pixel/action-space similarity and (b) matching XML/Accessibility button values. Agents may traverse the graph freely up to a step limit, accumulating rewards at key nodes, enabling offline evaluation with online-style path flexibility.

SMAN-Bench-Noisy sources 100 tasks from 20 heavily ad-contaminated third-party apps (static popups, dynamic video ads, redirect links), plus AITZ-Noise (2,504 trajectories each with one randomly injected ad). SMAN-Bench-Ambiguous strips slot information from full instructions to produce under-specified queries; agents may ask clarifying questions matched to preset Q&A pairs at corresponding GUI steps.

Key Contributions

  • GIAS: slot-based instruction annotation algorithm that maps unlabeled graph trajectories to templated natural-language instructions, supporting one-to-many trajectory matching
  • Offline multi-path evaluation: merges discrete trajectories into a key-node graph; agents earn step rewards by reaching equivalent key nodes rather than matching a single golden path
  • SMAN-Bench-Noisy: realistic advertisement-noise evaluation using real noisy apps and trajectory contamination; marked divergent steps support rollback-capable agents
  • SMAN-Bench-Ambiguous: active interaction evaluation where agents must issue clarification queries to fill missing slots; preset Q&A assigned to specific GUI steps
  • Cross-system coverage of iOS 18.5, HarmonyOS 5.0, and HyperOS 3.0 in addition to Android

Results

  • Common split (multi-path, AppAgent-v1 + GPT-4o): Step Acc = 61.8% (simple), 25.5% (complex); SR = 19.8% (simple), 25.5% (complex)
  • Common split (Qwen2-VL-72B): Step Acc = 62.8%/29.6%, SR = 20.6%/4.5% for simple/complex
  • Noisy split (GPT-4o, multi-path): Step Acc = 77.7%, SR = 30.5%
  • Noisy split (Qwen2-VL-72B, multi-path): Step Acc = 77.1%, SR = 29.5%
  • Complex tasks consistently show SR ~3–25% across all tested agents, exposing headroom
  • Only ~8% of complex-split data flagged as suboptimal in quality verification
  • Multi-path evaluation consistently yields higher SR than single-path, validating the multi-path design advantage

Limitations

  • Ambiguous and noisy splits are small (100 instructions each), limiting statistical power
  • The predefined query pool for search instructions means any out-of-pool search is invalid, potentially underestimating agent capability
  • GIAS relies on GPT-4o only for the final verification step; other stages use open-source models, which may introduce annotation noise
  • Graph coverage is bounded by Mobile3M's BFS random-walk exploration; rare app states are likely missing
  • Ambiguous Q&A interactions are preset rather than dynamically generated, which may not cover all agent clarification strategies
  • Results reported on a sampled Random-800 subset for cost reasons, not the full 12,856-instruction split

Relevance to Vision-Language Models

SMAN-Bench directly stresses VLMs in their role as the perception-and-reasoning backbone of GUI agents, exposing that even 70B+ models (Qwen2-VL-72B, GPT-4o) fail on complex multi-step mobile tasks with SR below 30%. The slot-based multi-path evaluation methodology is transferable to other VLM agentic benchmarks that currently over-constrain evaluation to single golden trajectories. The noisy split is particularly relevant for VLM robustness research, since advertisement overlays and pop-ups are adversarial visual distractors that corrupt GUI grounding. The ambiguous split additionally probes VLM pragmatic competence—knowing when to ask versus act—an under-studied dimension in existing VLM benchmarks.