MIP against Agent: Malicious Image Patches Hijacking Multimodal OS Agents¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper introduces Malicious Image Patches (MIPs), adversarially perturbed screen regions that hijack multimodal OS agents by exploiting their reliance on screenshots. When an OS agent captures a screen containing a MIP, it is induced to generate and execute a predefined malicious program via OS-level APIs. MIPs generalise across unseen user prompts, screen layouts, screen parsers, and multi-step execution contexts.

Problem¶

OS agents built on VLMs control computers directly through screenshots and API calls, vastly expanding the attack surface beyond text-only models. Existing adversarial attacks on VLMs target direct textual inputs (which can be filtered) or rely on overt visual elements like pop-ups; no prior work systematically investigates stealthy, patch-constrained image-domain attacks that operate through the full OS agent pipeline (screen parser → VLM → API executor) under realistic deployment constraints.

Method¶

MIPs are adversarial perturbations confined to a small patch region R (≈1/7 of the screen, e.g., a wallpaper or social-media image) that must survive the OS agent's multi-component pipeline. The attack formulates an objective to minimise cross-entropy between the VLM's output and a fully specified malicious target program y, subject to three constraints: (1) perturbation restricted to patch coordinates R with ℓ∞-norm ≤ ε; (2) the non-differentiable screen parser g must produce identical SOMs on both clean and perturbed screenshots (bounding boxes must not overlap R); (3) the non-differentiable resize function q is replaced with a differentiable approximation. Optimisation uses Projected Gradient Descent (PGD) with periodic constraint verification and rollback. Universal MIPs are obtained by jointly optimising over batches of (prompt, screenshot) pairs from 12 domains; multi-VLM MIPs jointly target 11B-IT, 90B-IT, and 11B-PT Llama-3.2 models. The attack encodes the complete malicious program directly into the perturbation, avoiding runtime assembly by the agent.

Key Contributions¶

Novel attack vector (MIPs) targeting OS agents via their screenshot capture mechanism, distinct from text-based prompt injection and pop-up attacks.
Formal treatment of the OS agent pipeline including screen parser non-differentiability, patch-region constraints, and integer pixel discretisation.
Demonstration of universal MIPs that transfer to unseen prompts, screen layouts, screen parsers (OmniParser → GroundingDINO+TesseractOCR), and intermediate execution steps.
First demonstration of an OS-agent computer worm concept: a MIP that causes an agent to share/post the MIP, enabling autonomous self-propagation.
Systematic evaluation in the Windows Agent Arena (WAA) across 154 tasks, two attack settings (desktop wallpaper, social media post), and two malicious behaviours.

Results¶

Targeted MIPs achieve ASR = 1.00 on the (prompt, screenshot) pair used for optimisation across both settings and temperatures.
Universal MIPs (optimised on P⁺ × S⁺): ASR = 1.00 on seen pairs; ASR ≥ 0.75–1.00 on fully unseen (P⁻ × S⁻) pairs at greedy decoding; ASR drops to 0.24–0.98 at temperature 1.0 depending on setting and target.
Parser transfer (OmniParser → GroundingDINO+TesseractOCR, unseen at optimisation): ASR = 0.29–0.98 on unseen pairs at temperature 1.0, and 0.40–0.62 at greedy decoding — substantial transfer without explicit cross-parser optimisation.
Execution-step transfer: ASR = 0.61–1.00 at temperature 0.0 and 0.38–0.72 at temperature 1.0 when MIP is encountered mid-task on seen and unseen screenshots.
Multi-VLM MIPs (11B-IT + 90B-IT + 11B-PT jointly optimised): ASR ≥ 0.92 on seen pairs and ≥ 0.93 on unseen pairs at temperature 0.0; fails to transfer to the unseen Llama-3.2-90B-Vision (pre-trained, not included in optimisation).
Targeted MIPs converge in 600–3,000 PGD steps; universal in 20,000–28,000; multi-VLM in ~74,000.

Limitations¶

MIPs must be encountered and captured by the OS agent — not guaranteed in practice.
Transferability across unseen VLMs (not included in optimisation) fails; cross-model generalisation remains an open problem.
Targeted MIPs do not transfer to unseen screenshots (ASR = 0.00), requiring universal optimisation for layout robustness.
Experiments are confined to open-source Llama-3.2 models within the WAA environment; generality to proprietary VLMs (GPT-4o, Gemini) is untested.
Crafting universal/multi-VLM MIPs is computationally expensive (~74,000 PGD steps for multi-VLM).
No defence evaluation is provided; proposed defences (stochastic augmentation, verifier modules) are speculative and may degrade benign task performance.

Relevance to Vision-Language Models¶

This paper directly probes a critical security gap that emerges when VLMs are deployed as embodied OS agents rather than passive text generators: the visual input pathway becomes an exploitation surface for persistent, patch-constrained adversarial attacks. The work extends classical PGD-based adversarial image methods into a richer pipeline involving screen parsing, multimodal context (SOMs + text memory), and downstream API execution, revealing that VLM alignment does not protect against carefully crafted visual perturbations at the OS-agent level. For researchers tracking VLMs, the multi-VLM universality results and the computer-worm propagation concept are particularly significant, as they anticipate security challenges that will scale with broader VLM deployment in agentic systems. The study also benchmarks how current VLM architectures (Llama-3.2 vision family) respond to adversarial visual inputs in long-context, multi-step decision-making settings.