Skip to content

SeerGuard: A Safety Framework for Mobile GUI Agents via World Model Prediction

🕒 Published (v1): 2026-07-17 01:45 UTC · Source: Arxiv · link

Why this paper was selected

World-model safety predictor for GUI agents before irreversible actions; directly applicable architecture for any action-space agent

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SeerGuard is a pre-execution safety framework for mobile GUI agents that uses a world model to predict the semantic consequences of proposed actions before they execute. It combines coarse-grained instruction-level screening with fine-grained action-level risk assessment via a Safety-Augmented World Model (SAWM) built on Qwen3-VL-8B-Instruct. On MobileSafetyBench, it improves the Safety-Utility Score (SUS) from 0.191 to 0.596 and cuts the Risk-Cost Score (RCS) from 0.347 to 0.130 for Qwen3-VL.

Problem

Mobile GUI agents operate on live interfaces where a single erroneous action (unintended purchase, data deletion, privacy leakage) is often irreversible. Existing defenses are either purely reactive—evaluating trajectories post-hoc—or purely instruction-level, failing when a benign-looking instruction becomes harmful only given the current GUI state. Neither paradigm supports consequence-aware pre-execution risk assessment.

Method

SeerGuard is a two-stage, cascaded guardrail:

  1. Instruction-level screening (\(G_\text{inst}\)): Before any interaction, a VLM-based filter classifies the user instruction as safe/unsafe with a rationale. Only clearly malicious instructions (e.g., "factory reset my phone") are rejected here; ambiguous ones are deferred to stage 2 to avoid over-refusal.

  2. Action-level risk assessment (\(G_\text{WM}\)): At each step \(t\), rather than predicting a pixel-level next screen, SeerGuard decodes a semantic next-state description \(\hat{s}_{t+1}\) via the factorization: $\(p(s_{t+1} | o_t, a_t) = \sum_{z_{t+1}} p(s_{t+1} | z_{t+1})\, p(z_{t+1} | o_t, a_t)\)$ where \(z_{t+1}\) is a latent semantic variable implicit in the VLM's hidden representations. The model outputs \((\hat{s}_{t+1}, l_\text{act}, r_\text{act})\): predicted next state, binary safety label, and rationale.

The SAWM is built by fine-tuning Qwen3-VL-8B-Instruct (1 epoch, lr=\(10^{-6}\)) on a 148K multi-task corpus combining: - MobileWorld next-state QA data (world model capability) - Re-annotated multimodal mobile trajectories (grounded safety) - 100-app synthetic text dataset (2,000 instances) bridging general safety concepts to GUI-specific risks

Training uses a 2:1 safe-to-unsafe ratio and joint SFT on both state-prediction and risk-classification objectives.

Key Contributions

  • SeerGuard framework: dual-stage (instruction + action level) pre-execution safety guardrail deployable over any black-box GUI agent without modifying the agent's policy.
  • Safety-Augmented World Model (SAWM): semantic (not pixel-level) next-state prediction jointly trained with risk classification, avoiding expensive image generation while preserving safety-relevant functional state changes.
  • Cross-modal safety augmentation strategy: three-tier data pipeline (general textual safety → multimodal mobile risk annotations → synthetic textual mobile risk) that sidesteps the infeasibility of collecting large-scale real-world risky GUI interactions.
  • MobileSafetyBench evaluation: demonstrated generalization across Qwen3-VL-8B, GPT-5.1, and Gemini-3.1 agents.

Results

All results on MobileSafetyBench (250 tasks: 150 high-risk, 100 low-risk):

  • Qwen3-VL + SeerGuard vs. Qwen3-VL baseline: RCS (\(\alpha=0.8\)) drops from 0.347 → 0.130; SUS (\(\omega=0.8\)) rises from 0.191 → 0.596.
  • GPT-5.1 + SeerGuard: RCS drops from 0.301 → 0.145; SUS rises from 0.573 → 0.679 (\(\alpha,\omega=0.8\)).
  • Gemini-3.1 + SeerGuard: RCS drops from 0.368 → 0.180; SUS rises from 0.581 → 0.773 (\(\alpha,\omega=0.8\)).
  • Ablation (Qwen3-VL): action-level only (SeerGuard\(_\text{act}\)) achieves RCS=0.141 / SUS=0.503, outperforming instruction-only (SeerGuard\(_\text{inst}\), RCS=0.310 / SUS=0.248), confirming world-model prediction is the dominant safety component.
  • SCoT baseline (safety-guided chain-of-thought prompting) does not improve RCS over vanilla Qwen3-VL (0.368 vs. 0.347) and barely moves SUS (0.219 vs. 0.191), confirming prompt-only approaches are insufficient.
  • Replacing SAWM with a raw Qwen3-VL guard (SeerGuard\(_\text{Qwen}\)) gives intermediate performance (RCS=0.170, SUS=0.554), validating the benefit of fine-tuning.

Limitations

  • Evaluated only on MobileSafetyBench (250 tasks); broader coverage of real-world app ecosystems not demonstrated.
  • SAWM is fine-tuned on Qwen3-VL-8B-Instruct and may not generalize without retraining to architecturally different base models.
  • The synthetic text-bridge dataset (2,000 LLM-generated instances across 100 apps) may not capture the full diversity of real-world risky interactions.
  • Pixel-level world models are explicitly excluded; semantic descriptions may miss visually encoded risks (e.g., amounts hidden in images, CAPTCHA-style UI elements).
  • Latency cost of running SAWM at every action step is acknowledged but not quantified in the provided text.
  • The 10-category safety taxonomy follows OS-Sentinel; coverage of novel or cross-category threats is not assessed.

Relevance to Agentic AI / LLM Agents

SeerGuard directly addresses the critical safety gap in autonomous agents that take real-world irreversible actions, shifting the paradigm from reactive trajectory auditing to proactive, consequence-aware pre-execution intervention—a fundamental requirement for deploying GUI agents at scale. The use of a semantic world model as a safety component is a novel integration of planning-oriented machinery (world models for foresight) with alignment-oriented goals (harm prevention), relevant to any LLM agent operating in stateful environments. The plug-in guardrail design—wrapping any black-box agent policy without modifying it—is practically significant for the ecosystem of diverse commercial GUI agents. The data augmentation strategy for safety-critical training data also speaks to a broader challenge: how to build safety classifiers for domains where collecting real-world risky demonstrations is ethically or legally prohibited.