Skip to content

Agent-SAMA: State-Aware Mobile Assistant

🕒 Published (v1): 2025-05-29 16:08 UTC · Source: Arxiv · Venue: AAAI · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Agent-SAMA is a mobile GUI agent framework that models app navigation as a Finite State Machine (FSM), treating UI screens as states and user actions as transitions. Four specialized agents collaboratively construct FSMs in real time to guide planning, execution, verification, and error recovery. This structured state representation yields up to 12% improvement in task success rate and 13.8% in recovery success rate over prior reactive baselines.

Problem

Existing mobile GUI agents are fundamentally reactive: they select the next action based solely on the current screen without maintaining any structured model of app navigation flow. This prevents agents from tracking execution context, detecting unexpected state transitions, and recovering from errors in a principled way—limiting performance on complex, multi-step, cross-app tasks.

Method

Agent-SAMA defines each task execution as traversal over a Finite State Machine \(M = (S, A, T, s_0, G)\), where \(S\) is the set of discovered UI states (each a natural-language description of one screen), \(A\) is the action space, and transitions \(T(s_i, a_i) = (s_{i+1}, pre_{i+1}, post_i)\) carry pre- and post-conditions. Four agents collaborate across four phases:

  1. Planning: Planner generates \(n=5\) candidate plans \(\Pi = \{\pi_1,\ldots,\pi_5\}\), then an LLM-as-judge selects the best one against a rubric (goal relevance, efficiency, robustness, clarity).
  2. Execution: Screen Parser (OCR via DBNet + ConvNextViT, icon grounding via GroundingDINO, captions via Qwen-VL-Plus) extracts UI elements; State Agent incrementally builds the FSM using state beacons (concise semantic labels) to merge duplicate nodes; Actor Agent performs the selected action.
  3. Verification & Recovery: Reflection Agent compares the FSM-predicted post-condition against the actual new screen; on NoChange or Fail, it identifies a previously verified stable state \(s_j\) and constructs a backward recovery plan \(\pi_r = [a_{i-1}, a_{i-2}, \ldots]\); after \(n=2\) repeated failures, replanning is triggered.
  4. Knowledge Retention: Mentor Agent extracts reusable knowledge \(K\) (action sequences with preconditions, guidance cues, and full FSMs) stored in long-term memory and injected as context for future tasks.

Cross-app tasks maintain separate FSMs per app. Backbone: GPT-4o-2024-11-20 at temperature 0.

Key Contributions

  • First incorporation of FSM modeling into mobile GUI agents, treating each app as a formal state machine constructed online during execution.
  • State Agent with pre/post-conditions on every state and transition, enabling proactive verification rather than purely reactive action selection.
  • LLM-based judge for multi-candidate plan selection and replanning, reducing suboptimal single-path planning.
  • Mentor Agent for cross-task knowledge retention: stores FSMs, action sequences, and guidance cues in persistent memory for reuse.
  • Demonstrated model-agnostic architecture (evaluated across GPT-4o, Claude 3.5, Gemini 1.5 Pro backbones).

Results

Mobile-Eval-E (25 tasks, 15 apps, avg. 14.56 actions/task, physical Pixel 7 Pro): - SR: 84.0% (+12% vs. Mobile-Agent-E+Evo at 72.0%) - SS: 86.15% (+7.18%) - AA: 83.24% (+6.59%) - TR: 16.0% (↓8% from baseline's 24.0%) - RS: 71.88% (+4.53%) - Fewer errors: 32 vs. 49

SPA-Bench (20 English cross-app tasks, 25 apps): - SR: 80.0% (+5% vs. baseline's 75.0%) - SS: 88.64% (+8.33%) - AA: 84.35% (+6.49%) - RS: 66.67% (+13.81%) - Fewer errors: 63 vs. 70

AndroidWorld (116 task templates, 20 apps, emulator): - SR: 63.7%, outperforming AgentS2 (54.3%), V-Droid (59.5%), Mobile-Agent-E+Evo (53.4%)

Backbone ablation (Mobile-Eval-E SR): GPT-4o 84.0% > Claude 3.5 75.0% > Gemini 1.5 Pro 68.0%.

Limitations

  • Evaluated on at most 25 cross-app tasks (Mobile-Eval-E); small task count limits statistical confidence.
  • AndroidWorld is ~90% single-app, so cross-app generalization at scale is not fully tested.
  • FSM state explosion is mitigated by beacons but not formally bounded; performance on very long-horizon tasks with large state spaces is unclear.
  • Recovery plan execution terminates after only \(n=2\) repeated failures before replanning; threshold is fixed and not adaptive.
  • Backbone dependency: substantial performance gap between GPT-4o and weaker models (16% SR gap to Gemini 1.5 Pro on Mobile-Eval-E) suggests the framework benefits disproportionately from stronger MLLMs.
  • Human evaluation on cross-app benchmarks introduces inter-rater variability, despite a tiebreaker protocol.

Relevance to Harnesses / Meta-Harnesses

Agent-SAMA is a direct instance of a multi-agent harness architecture: it coordinates five specialized sub-agents (Planner, Screen Parser, State Agent, Actor, Reflection/Recovery, Mentor) within a structured four-phase execution pipeline, exactly the kind of orchestration layer that meta-harness research targets. The FSM serves as a shared, structured world-model that all agents read and write, functioning as a lightweight state-management substrate analogous to harness-level context stores. The LLM-as-judge planning module and the Mentor's cross-task knowledge base are reusable harness primitives—candidate plan selection and persistent memory injection—that could be lifted into general agentic orchestration frameworks. For researchers tracking harness design, Agent-SAMA concretely demonstrates how formalizing execution state (pre/post-conditions, FSM transitions) at the harness level significantly improves both success rates and error recovery compared to purely prompt-driven reactive loops.