Skip to content

GhostEI-Bench: Do Mobile Agent Resilience to Environmental Injection in Dynamic On-Device Environments?

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Benchmark for mobile agent resilience to environmental injection in dynamic interfaces

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

GhostEI-Bench is the first benchmark specifically for evaluating mobile VLM agents against dynamic environmental injection attacks—adversarial UI elements (overlays, spoofed notifications) injected at runtime into live Android emulators. Evaluation of 8 state-of-the-art VLM agents reveals that most exhibit Vulnerability Rates (VR) in the 40–55% range, with GPT-5 as the notable exception at 16.43% VR. The benchmark exposes a critical security gap: visual safeguards in current agents are systematically insufficient against real-time UI manipulation.

Problem

Prior mobile agent security benchmarks (MobileSafetyBench, MLA-Trust, MMBench-GUI) evaluate agents against static UI states or predefined textual threats. They lack coverage of dynamic environmental injection: adversarial UI elements injected at runtime—pop-ups, spoofed system overlays, deceptive SMS notifications—that bypass textual safeguards by corrupting the agent's visual perception mid-task. No prior work provides a unified, reproducible, executable-environment framework for systematically quantifying this threat.

Method

GhostEI-Bench runs inside fully operational Android emulators (14 apps: 9 native, 5 third-party). A hook-based trigger mechanism intercepts a predefined agent action (e.g., launching a target app) via ADB broadcast, then a custom on-device helper renders adversarial UI elements in real-time. Three attack vectors are evaluated: 1. Deceptive Instruction: inherently harmful user prompts testing foundational safety alignment. 2. Static Environmental Injection: sensitive data pre-seeded in the environment (e.g., passwords in notes) that the agent should not exfiltrate. 3. Dynamic Environmental Injection: runtime overlay or popup SMS events timed to interrupt the agent at critical decision points.

Test cases (110 total) are generated via LLM then manually vetted across 7 domains Ă— 7 risk fields. Each execution is assessed by an LLM judge that receives the full action trajectory plus screenshot sequence and classifies outcomes as TC, FAS, PAS, or BF (Benign Failure). The primary security metric is:

\[\text{VR} = \frac{\text{Count(FAS)} + \text{Count(PAS)}}{\text{Total Cases} - \text{Count(BF)}}\]

which isolates security susceptibility from raw capability failures.

Key Contributions

  • Formalizes environmental injection as a qualitatively distinct adversarial threat model for mobile agents, separate from jailbreak and static GUI attacks.
  • Releases GhostEI-Bench: 110 test cases spanning 3 attack vectors, 7 domains, 7 risk fields, executed in live Android emulators with a hook-based real-time injection engine.
  • Proposes a trajectory-aware LLM judge evaluation protocol that pinpoints failure locus (perception, recognition, or reasoning) rather than just pass/fail.
  • Empirically evaluates 8 VLMs across two agent frameworks (Mobile-Agent-v2, AppAgent) and two specialized GUI models (UI-TARS variants).

Results

  • GPT-5 achieves the lowest VR at 16.43% with 56.4% TC—far more robust than all other tested models.
  • Most models cluster in the 40–55% VR range:
  • Claude-3.7-Sonnet: 55.12% VR, 33.6% TC
  • GPT-4o (Mobile-Agent-v2): 54.87% VR, 34.6% TC
  • UI-TARS-7B-SFT: 56.25% VR (highest overall vulnerability), 26.4% TC
  • Gemini-2.5 Pro: 40.00% VR, 50.0% TC
  • GPT-5-chat-latest (preview): 41.67% VR, 45.5% TC
  • AppAgent framework generally yields similar or slightly worse VR versus Mobile-Agent-v2 on the same backbone.
  • UI-TARS (domain-specific fine-tuned models) does not confer security benefits; UI-TARS-7B-SFT has the worst VR in the benchmark.
  • Privacy Leakage (67 cases) and Fraud (43 cases) are the most prevalent risk categories; Settings (60) and Productivity (46) are the most frequent domains.

Limitations

  • Benchmark size is small (110 test cases), which may limit statistical power for fine-grained comparisons.
  • Dynamic injection currently covers only two event types (Overlays, Popup SMS); other real-world vectors (e.g., accessibility service abuse, inter-app data channels) are not modeled.
  • The LLM judge's reliability is assumed but not independently validated against human annotation at scale.
  • Claude-Sonnet-4 (preview) exhibited localization/positional bias during testing, potentially confounding its results.
  • Mitigation strategies are not explored beyond testing self-reflection and reasoning modules; no defenses are proposed.
  • Emulator-based execution may not fully capture device-specific or OS-version-specific behaviors.

Relevance to Agentic AI / LLM Agents

GhostEI-Bench directly targets a known weakness of VLM-based agents operating in open-ended environments: the assumption that visual perception is trustworthy. As agents gain broader GUI-level autonomy—executing financial transactions, accessing private data, managing communications—the threat surface from adversarial environmental manipulation grows proportionally. The benchmark's trajectory-aware LLM judge methodology (failure attribution across perception/recognition/reasoning) provides a reusable evaluation pattern applicable beyond mobile settings, including desktop GUI agents and web agents. The finding that domain-specific fine-tuning (UI-TARS) does not improve security robustness is a significant negative result that challenges the assumption that task specialization implicitly improves alignment.