Skip to content

MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments

🕒 Published (v1): 2025-12-22 14:31 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MobileWorld is a mobile-agent benchmark of 201 tasks across 20 apps that extends AndroidWorld with long-horizon cross-app workflows, agent-user interaction (ask_user clarification dialogues), and MCP-augmented hybrid tool-use tasks. It pairs these novel task categories with a fully deterministic, containerized evaluation infrastructure. Best results: 51.7% SR (agentic framework with GPT-5), 20.9% SR (best end-to-end model), versus 90%+ on the now-saturated AndroidWorld.

Problem

AndroidWorld is saturating (SOTA >90% SR), lacks e-commerce/enterprise app categories, assumes fully-specified instructions, and ignores hybrid GUI+external-tool execution. No existing mobile benchmark covers agent-user clarification dialogues or MCP tool invocation alongside GUI actions, and commercial-app benchmarks cannot achieve deterministic evaluation due to opaque backends.

Method

MobileWorld defines three task types within a POMDP formulation: (1) GUI-Only tasks (standard mobile UI manipulation and information retrieval); (2) Agent-User Interaction tasks, where critical information is deliberately withheld and agents must invoke an ask_user action routed to a GPT-4.1-simulated user agent holding the missing context; (3) MCP-Augmented tasks requiring hybrid invocation of 64 tools across 5 MCP servers (Amap Maps, GitHub, Jina AI, Stockstar, arXiv) interleaved with GUI operations. The evaluation environment runs inside Docker-in-Docker with a rooted AVD, self-hosted open-source app backends (Mattermost, Mastodon, Mall4Uni), snapshot-based state management, and four deterministic verification modes: textual pattern matching, backend database queries, ADB local storage inspection, and application callbacks. A planner-executor agentic framework is provided as a baseline: a VLM planner issues natural-language element descriptions; a grounding model (UI-Ins-7B) converts them to pixel coordinates; ask_user and mcp_call are first-class action primitives injected into the shared interaction history.

Key Contributions

  • 201-task benchmark with 27.8 average completion steps (vs. 14.3 for AndroidWorld) and 62.2% cross-app tasks (vs. 9.5%).
  • First mobile benchmark with agent-user interaction tasks (22.4% of tasks): ask_user action + LLM-simulated user agent pattern.
  • First mobile benchmark with MCP-augmented tasks (19.9% of tasks): 64 tools across 5 servers, hybrid GUI+MCP execution.
  • Fully deterministic evaluation via self-hosted open-source backends, eliminating MLLM-as-a-judge noise.
  • New metrics: UIQ (User Interaction Quality, penalizes over/under-querying), Ave. Queries, Ave. MCP Calls.
  • Planner-executor agentic framework extending action space with ask_user and mcp_call primitives.
  • Characterizes five open research challenges: ambiguity detection, MCP context management, long-term memory/state checking, complex logic reasoning, spatial-temporal context awareness.

Results

  • Best agentic framework (GPT-5 + UI-Ins-7B): 51.7% overall SR; 54.0% GUI-Only, 62.2% User-Interaction, 51.6% MCP.
  • Best end-to-end model (Doubao-1.5-UI-TARS): 20.9% overall SR, 26.3% GUI-Only.
  • End-to-end models score near 0% on MCP-augmented tasks and 0–6.7% on agent-user interaction tasks (models without native ask_user support excluded).
  • Qwen3-VL-235B-A22B achieves 9.5% overall as end-to-end; with agentic framework structure, GPT-5 gains +30pp vs. best end-to-end.
  • Sharp drop vs. AndroidWorld SOTA (~90%+): best framework achieves only 51.7% on MobileWorld.

Limitations

  • Self-hosted open-source app substitutes (Mattermost for Slack, Mastodon for Twitter) may not fully replicate the UX complexity of their commercial counterparts.
  • MCP servers are curated from a single platform (Alibaba Bailian); tool diversity is limited to 5 servers/64 tools.
  • Simulated user agent (GPT-4.1) may not capture the full variability of real human clarification responses.
  • Benchmark is Android-only; no iOS coverage.
  • Maximum 50-step evaluation cap may not expose all failure modes of long-horizon tasks.
  • Task set size (201) is modest; statistical power for fine-grained capability distinctions is limited.

Relevance to Harnesses / Meta-Harnesses

MobileWorld instantiates a multi-component evaluation harness in which an orchestrating framework routes agent actions across three execution channels—GUI ops, LLM-simulated user dialogue, and MCP server calls—then routes verification across four deterministic backends, making it a concrete example of a domain-specific meta-harness. Its planner-executor framework with first-class ask_user and mcp_call primitives demonstrates how harness architecture shapes what agent capabilities can be measured and exercised. The Docker-in-Docker containerization with snapshot-based reset and multi-modal verification (DB, ADB, callbacks, pattern match) illustrates infrastructure patterns relevant to building reproducible harnesses for agentic systems. Researchers building harnesses for tool-augmented or user-interactive agents can directly adopt its action-space extension and evaluation-routing patterns.