Skip to content

SMAN-Bench: A Cross-System Benchmark for Mobile Agents under Single- and Multi-path, Ambiguous, and Noisy Tasks

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SMAN-Bench is a cross-system mobile-agent benchmark addressing the dual failure modes of existing offline (single-path, ignores multi-solution nature) and online (unstable step rewards from app/OS updates) evaluations. It introduces 12,856 instructions across Common, Noisy, and Ambiguous splits, evaluated under both single- and multi-path regimes on four OS platforms. The GIAS pipeline auto-generates slot-filled instructions from an unlabeled graph corpus without human annotation, enabling stable, fine-grained step rewards.

Problem

Offline mobile-agent benchmarks enforce a single golden trajectory, penalizing valid alternative solutions; agents that overfit these single-path annotations perform poorly in real deployments. Online benchmarks suffer from non-deterministic step rewards caused by OS/app updates and user-state drift. No existing benchmark systematically tests agents under realistic noise (ads, pop-ups, redirect pages) or ambiguous (underspecified) instructions requiring proactive clarification.

Method

Data corpus. Built on Mobile3M, a directed graph of 3.09M UI pages and ~20M interactions from 49 apps, where nodes are screenshots+XML and edges are transition actions.

GIAS (Generating Instructions From Action Sequences). A six-stage pipeline: 1. Sample multi-path trajectories between homogeneous start/end nodes. 2. Annotate GUI page descriptions via VLM. 3. Infer action intent from consecutive page-description pairs and the transition action. 4. Extract slot values (e.g., product name, SKU, artist) from GUI changes. 5. Fill 800 hand-authored task templates with extracted slots to produce instructions. 6. Deduplicate via semantic similarity threshold \(\tau\).

Only the final verification step uses a closed-source model; all other steps use open-source VLMs.

Multi-path evaluation. Individual trajectories sharing equivalent key nodes (matched by BM25 pixel similarity and XML button-value consistency) are merged into a graph. An agent can navigate freely within the graph and accumulates step rewards when it reaches key nodes, regardless of which path it took. Single-path evaluation retains the classic action-matching mode.

Noisy split. 100 instructions from 20 heavily ad-laden apps (static pop-ups, dynamic video ads, redirect links) plus AITZ-Noisy (2,504 AITZ trajectories each with one injected ad from a 150+ ad pool). Rollback markers are included.

Ambiguous split. Full instructions are reduced to underspecified forms by removing slot values; preset Q&A pairs are anchored to the GUI step where clarification is needed, enabling interactive evaluation.

Platforms. Android (primary), iOS 18.5, HarmonyOS 5.0, HyperOS 3.0.

Key Contributions

  • SMAN-Bench: 12,856 instructions (12,854 Common + 100 Noisy + 100 Ambiguous), bilingual EN/CN, 15 categories, 49 apps, avg 7.28 steps, ~48k screenshots.
  • GIAS: fully automated slot-based instruction-generation pipeline from unlabeled graph corpora; 800 templates; only verification requires a closed-source model.
  • Offline multi-path evaluation: graph-merging of trajectories on key nodes enables stable step-reward signals without live device execution.
  • SMAN-Bench-Noisy: realistic noise benchmark with at least 5 noisy steps per trajectory (ads, redirects, auth screens); also AITZ-Noisy contamination variant.
  • SMAN-Bench-Ambiguous: proactive-interaction benchmark where agents must issue clarification questions to receive slot information progressively.
  • Evaluation of 20+ models including reasoning VLMs (o3, Claude-3.7-Sonnet, QVQ, DeepSeek-R1) and RL-trained mobile agents (UI-Tars, GUI-R1, UI-R1).

Results

  • Best Common-Simple SR: Mobile-Agent-E + Qwen2-VL-72B (single-path 32.5%; multi-path 29.5%, SE=3.7).
  • Best Common-Simple multi-path SR: Mobile-Agent-E + GPT-4o (30.5%, SE=3.9), slightly below Qwen2-VL-72B single.
  • Common-Complex is dramatically harder: best observed SR in the truncated table is ~25.5% (Mobile-Agent-v2 + GPT-4o multi-path), most models below 10%.
  • Mobile-Agent-E consistently outperforms AppAgent-v1 and Mobile-Agent-v2 across VLMs; e.g., with LLAVA-72B-NEXT, SR jumps from 0%/15.1% to 18.4%.
  • LLAVA-72B-NEXT + AppAgent: near-zero SR on Common-Simple despite 93.2% TYPE accuracy, indicating action-type correctness does not predict task success.
  • GPT-4o Step.Acc peaks at 77.7% (Mobile-Agent-E multi-path, Common-Simple) while SR remains ~30%, showing the gap between step correctness and end-to-end task completion.
  • ~8% of complex-split instructions flagged as suboptimal during quality verification (excess redundant steps from multi-slot templates).

Limitations

  • Noisy and Ambiguous splits are small (100 instructions each), limiting statistical power.
  • Multi-path evaluation requires a predefined query pool; any search result outside the pool is declared invalid, constraining search-heavy tasks.
  • Graph corpus (Mobile3M) is Android-centric; iOS/HarmonyOS/HyperOS coverage added but not quantified separately.
  • Template-based instruction generation (800 templates for 12,854 instructions) may introduce distributional artifacts from template reuse.
  • Ambiguous split Q&A is pre-scripted, not dynamically generated, so agent clarification strategies that diverge from the preset question order may not be properly credited.

Relevance to Harnesses / Meta-Harnesses

SMAN-Bench is itself a benchmark harness that wraps three agent frameworks (AppAgent-v1, Mobile-Agent-v2, Mobile-Agent-E) under a unified evaluation protocol, making it directly relevant as a meta-evaluation layer over multi-agent systems. The GIAS pipeline is a data-generation sub-harness that ingests an unlabeled graph corpus and emits structured (instruction, trajectory, slot, Q&A) tuples with minimal human intervention — an archetypal automated pipeline component. The multi-path graph evaluation engine demonstrates how a harness can decouple execution mode (single vs. graph traversal) from reward computation, a separation of concerns central to composable meta-harness design. For researchers building evaluation or orchestration harnesses for GUI agents, SMAN-Bench's slot-key-node abstraction provides a reusable primitive for stable, path-agnostic step rewards.