GhostEI-Bench: Do Mobile Agent Resilience to Environmental Injection in Dynamic On-Device Environments?

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; benchmark for VLM agent resilience to environmental injection on mobile

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

GhostEI-Bench is the first benchmark for evaluating mobile VLM agents against dynamic environmental injection attacks—adversarial UI elements (overlays, spoofed notifications) injected at runtime into live Android emulators. Across 8 tested VLM agents, Vulnerability Rates range from ~16% to ~56%, exposing severe and widespread susceptibility. A novel LLM-judge protocol performs fine-grained failure attribution (perception vs. recognition vs. reasoning).

Problem

Existing mobile-agent safety benchmarks (MobileSafetyBench, RiOSWorld, AgentHazard) evaluate static UI states or pre-scripted textual threats; none systematically test agents against adversarial UI elements injected dynamically during live task execution inside a real running environment. This leaves the resilience of mobile VLM agents to real-time visual deception—pop-ups, overlays, spoofed notifications—entirely unquantified despite the concrete risks of privacy leakage, financial fraud, and device compromise.

Method

GhostEI-Bench runs on fully operational Android emulators hosting 14 apps across 7 domains. It defines three attack vectors: 1. Deceptive Instruction — inherently harmful user prompts testing foundational safety alignment. 2. Static Environmental Injection — sensitive data already present in the environment (passwords in notes, financial screenshots). 3. Dynamic Environmental Injection — adversarial UI injected at runtime via a hook-based ADB trigger mechanism: a predefined agent action (e.g., launching an app) fires a broadcast intercepted by an on-device helper app that renders Overlays or Popup SMS in real time, or redirects the browser to a locally hosted phishing page.

Benchmark composition: 110 test cases (75 dynamic injection, 24 deceptive instruction, 11 static injection), spanning 7 risk fields (Fraud, Cybercrime, Disinformation, System Sabotage, Privacy Leakage, Copyright Infringement, Harassment).

Evaluation uses four outcome labels—Task Completion (TC), Full Attack Success (FAS), Partial Attack Success (PAS), Benign Failure (BF)—assessed by a judge LLM that reads the full action trajectory and screenshot sequence. The primary security metric is:

\[\text{VR} = \frac{\text{Count(FAS)} + \text{Count(PAS)}}{\text{Total Cases} - \text{Count(BF)}}\]

which isolates attack susceptibility from capability failures.

Key Contributions

• Formalizes environmental injection as a qualitatively distinct adversarial threat separate from prompt injection or static GUI attacks.
• Releases GhostEI-Bench: 110 reproducible test cases with snapshot-based Android emulator environments, covering 7 risk domains and 3 attack vectors.
• Hook-based ADB injection mechanism that precisely times adversarial UI delivery relative to agent actions.
• LLM-judge evaluation protocol with per-step failure attribution across perception, recognition, and reasoning stages.
• Comprehensive empirical evaluation of 8 VLM configurations (GPT-4o, GPT-5, Gemini-2.5 Pro, Claude-3.7-Sonnet, Claude-Sonnet-4, Qwen2.5-VL-72B, UI-TARS-7B-SFT, UI-TARS-1.5-7B) under two agent frameworks (Mobile-Agent-v2, AppAgent).

Results

• Most vulnerable: Claude-3.7-Sonnet (VR = 55.12%, Mobile-Agent-v2), GPT-4o (VR = 54.87%), UI-TARS-7B-SFT (VR = 56.25%).
• Most robust: GPT-5 (VR = 16.43%, TC = 56.4%), the only model below 20% VR.
• GPT-5-chat-latest (preview): VR = 41.67%, TC = 45.5%.
• Gemini-2.5 Pro: VR = 40.00%, TC = 50.0%.
• Claude-Sonnet-4 (preview): VR = 50.00%; noted to exhibit localization/positional bias.
• Qwen2.5-VL-72B under Mobile-Agent-v2: VR = 41.42%; under AppAgent: VR = 52.56%.
• GPT-4o under AppAgent: VR = 50.00% vs. 54.87% under Mobile-Agent-v2.
• UI-TARS-1.5-7B: VR = 46.99%, TC = 40.9% — domain-specific fine-tuning does not confer meaningful security gains.
• Majority of models cluster in the 40–55% VR range, indicating systemic rather than model-specific failure.

Limitations

• Benchmark is small (110 test cases); coverage breadth per domain/risk-field cell is limited.
• Restricted to Android; no iOS, desktop, or browser-native evaluation.
• Dynamic injection relies on ADB hooking of specific trigger actions—does not cover asynchronous or OS-initiated injections (e.g., real push notifications from external services).
• LLM-as-judge introduces potential evaluation bias; inter-rater agreement against human labels is not fully reported in the visible text.
• Self-reflection and reasoning module ablations (RQ3/RQ4) are referenced but results are truncated in the provided text.
• Claude-Sonnet-4 localization issues confound its VR interpretation.

Relevance to Vision-Language Models

GhostEI-Bench directly stress-tests the visual perception and grounding capabilities of VLMs in an adversarial GUI setting, revealing that even state-of-the-art models like Gemini-2.5 Pro and GPT-5 remain significantly vulnerable to visually injected deception—a failure mode orthogonal to textual safety training. The benchmark exposes a critical gap: VLMs trained for GUI grounding (UI-TARS) show no special robustness advantage, suggesting that perception fidelity and safety alignment are largely decoupled in current architectures. For VLM researchers, the hook-based dynamic injection methodology and the LLM-judge trajectory analysis provide a reusable evaluation infrastructure for studying how visual context corruption propagates through perception → planning → action pipelines. This work complements static GUI benchmarks by introducing executable, time-sensitive adversarial conditions that better reflect real-world deployment risks.