Skip to content

Mobile-Agent-RAG: Driving Smart Multi-Agent Coordination with Contextual Knowledge Empowerment for Long-Horizon Mobile Automation

🕒 Published (v1): 2025-11-15 15:22 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Mobile-Agent-RAG augments a hierarchical Manager–Operator multi-agent framework with dual-level RAG: Manager-RAG retrieves human-validated high-level task plans to curb strategic hallucinations, while Operator-RAG retrieves app-specific (subtask, screenshot, action) examples to ground atomic UI actions. On the introduced Mobile-Eval-RAG benchmark, the system improves task completion rate by 11.0% and step efficiency by 10.2% over the strongest prior baseline.

Problem

MLLM-based mobile agents fail on long-horizon, cross-application tasks because they rely entirely on static internal knowledge, producing (1) strategic hallucinations during multi-step planning and (2) incorrect atomic actions during UI execution. Hierarchical decoupling of planning from execution (e.g., Mobile-Agent-E) alleviates the first but does not inject external grounding at either level, leaving hallucination as the root cause of cascading errors.

Method

The framework inherits the Manager–Operator hierarchy of Mobile-Agent-E and adds two RAG modules with dedicated knowledge bases.

Manager-RAG: At \(t=1\), the Manager Agent receives a retrieved set \(R_M\) of top-\(k\) documents from \(K_{MR}\)—pairs of (task instruction \(I_{MR}\), human-annotated operation steps \(H_{MR}\))—selected by cosine similarity via Contriever-MSMARCO embeddings. These serve as few-shot examples: \(P = \text{PromptGen}_M(I_{\text{query}}, R_M)\).

Operator-RAG: Before each atomic action, the Operator queries a per-app knowledge base \(K_{OR}^{\text{app}}\) with the current subtask \(T_{\text{query}}^{\text{app}}\), retrieving the single most similar (subtask, screenshot, action) document \(R_O\) to condition action generation: \(A = \text{PromptGen}_O(T_{\text{query}}^{\text{app}}, S_{\text{current}}, R_O)\).

Supporting modules—Perceptor (screenshot→structured visual info \(V_t\)), Action Reflector (before/after comparison for error detection), and Notetaker (persistent cross-step state \(N_t\))—complete the iterative loop. \(K_{MR}\) is fully human-curated from Mobile-Eval-RAG traces; \(K_{OR}^{\text{app}}\) is semi-automated (logged during execution, then human-verified) with per-app partitioning to avoid cross-app retrieval interference.

Key Contributions

  • Mobile-Agent-RAG: hierarchical multi-agent framework with dual-level RAG (Manager-RAG for planning, Operator-RAG for execution).
  • Two specialized retrieval KBs: \(K_{MR}\) (task→human-step plans) and \(K_{OR}^{\text{app}}\) (per-app subtask/screenshot/action triples).
  • Mobile-Eval-RAG benchmark: 50 cross-application long-horizon tasks averaging 16.9 steps across 2–3 apps, spanning five task categories.

Results

All numbers vs. Mobile-Agent-E (Gemini-1.5-Pro backbone) on Mobile-Eval-RAG multi-app tasks unless noted:

  • Completion Rate (CR): 58.3% → 75.7% (+17.4 pp); +Evo baseline is 61.2%.
  • Operator Accuracy (OA): 74.1% → 90.1%.
  • Step Efficiency: 2.60 → 4.03 (higher is better).
  • Success Rate (SR): 48.0% → 76.0%.
  • Steps: 22.4 → 18.8 (fewer steps needed).
  • GPT-4o backbone: SR 60% → 84%, CR 67.0% → 79.1%.
  • Claude-3.5-Sonnet: SR 60% → 84%, CR 72.3% → 79.6%.
  • Simple tasks (CR): 63.4% (Mobile-Agent-E) → 78.0% (RAG); SR 80% → 90%.
  • Complex tasks (CR): 54.9% → 74.2%; SR 26.7% → 66.7%.

Limitations

  • Knowledge bases require manual human annotation, creating a significant data-collection bottleneck for new apps or domains.
  • Evaluation covers only 50 benchmark tasks; statistical significance of per-metric gains is not reported.
  • Operator-RAG uses top-1 retrieval; in novel UI states with no near-neighbor in \(K_{OR}^{\text{app}}\), grounding degrades to the base model.
  • KBs are static at inference time; there is no online updating from successful agent executions during deployment.
  • Generalization to apps absent from \(K_{OR}^{\text{app}}\) is untested; per-app partitioning assumes the target app was seen during KB construction.

Relevance to Harnesses / Meta-Harnesses

Mobile-Agent-RAG is a concrete instantiation of a knowledge-augmented agentic harness: it orchestrates five distinct agent/module roles (Manager, Operator, Perceptor, Reflector, Notetaker) under a deterministic iterative loop, injecting retrieved external context at two separate abstraction layers rather than relying on a monolithic LLM call. The dual-level RAG injection pattern—different retrieval granularity at planning vs. execution layers—demonstrates how harness designers can route different knowledge types to appropriate sub-agents, a generalizable architectural principle. The semi-automated KB construction pipeline (logged execution → human verification → partitioned retrieval store) is directly reusable in any agentic harness that needs to bootstrap from human demonstrations. Compared to memory-only harnesses, the separation of retrieval corpus from agent working memory offers cleaner control over what knowledge each agent layer can access.