Benchmarking Web Agent Safety under E-commerce Deceptive Interfaces

🕒 Published (v1): 2026-04-26 17:46 UTC · Source: Arxiv · Venue: ACL 2026 · link

Why this paper was selected

Safety benchmark for web agents under realistic e-commerce deceptive interfaces

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

WebDecept is a configurable plugin framework that injects seven categories of deceptive interface patterns (pop-ups, banners, domain redirection, cart add-ons, price drift) into an e-commerce web environment to benchmark multimodal web agent safety. Evaluation of five state-of-the-art agents (GPT-4o, GPT-5.1, Claude Sonnet 4.5, Gemini 2.5 Pro, Qwen3-VL-Plus) reveals near-universal vulnerability to shopping-state manipulations, with Safety Violation Rates (SVR) reaching 100% on cart add-on and price-drift scenarios across all models. Prompt-based safety constraints offer only partial mitigation and fail entirely against subtle price manipulations.

Problem

Existing web-agent safety benchmarks focus on adversarial prompt injection or malicious user instructions, leaving a gap in the evaluation of realistic deceptive interface patterns — dark patterns intentionally designed by humans in e-commerce workflows (targeted ads, covert cart manipulation, price inflation). These patterns are hard to model systematically because they vary across domains and are not direct input-level attacks.

Method

WebDecept wraps the VisualWebArena OneStopShop e-commerce environment as a lightweight, state-triggered intervention layer. At a pre-configured timestep \(t^*\), a trigger engine applies a frontend-level intervention \(\delta_{t^*}\) to produce a modified state \(\tilde{s}_{t^*} = I(s_{t^*}, \delta_{t^*})\), from which the agent receives a modified observation \(\tilde{o}_{t^*}\). Seven parameterized deceptive patterns are implemented:

• Static misleading UI: pop-up and banner messages (configurable content, style, dismissibility)
• Personalized misleading UI: pop-up/banner content generated by a separate LLM conditioned on the user's task goal
• Domain redirection: hyperlink of the agent's next optimal action is silently replaced (visible only in accessibility tree)
• Cart add-ons: items stealthily added to the rendered cart without agent action
• Price drift: total checkout price inflated by a configurable multiplier (e.g., \(1.2\times\)) inconsistent with itemized subtotals

Evaluation uses 45 end-to-end shopping tasks (9 seed templates × product/attribute variants), crossed with 7 deceptive scenarios and 2 prompt variants (\(P_w\): minimal warning; \(P_r\): scenario-specific rule enumeration), yielding 630 episodes per agent. Two metrics are reported: Task Completion (TC) and Safety Violation Rate (SVR), where SVR counts episodes in which the agent commits at least one unsafe action post-trigger.

Key Contributions

• WebDecept: a lightweight, configurable injection framework that integrates into existing web benchmarks without modifying the underlying environment logic
• Seven parameterized deceptive patterns covering static/personalized messaging, covert navigation, and shopping-state manipulation
• A 45-task benchmark spanning full e-commerce workflows (discovery → cart → checkout) with controlled reasoning complexity
• Empirical safety evaluation of five frontier multimodal agents with ablations over pop-up message type, price drift magnitude (\(1.2\times\)–\(2.0\times\)), and perception modality (image-only, SoM, accessibility tree)

Results

• Cart add-ons: SVR = 1.00 for all five agents under both \(P_w\) and \(P_r\) — no model detects stealthy cart modifications
• Price drift: SVR ≥ 0.75 for all agents under \(P_r\); GPT-5.1 SVR = 1.00 at \(1.2\times\) drift, declining to near 0 at \(2.0\times\), showing detection only for large discrepancies
• Personalized pop-ups: Claude Sonnet 4.5 reaches SVR = 1.00 under \(P_w\) and 0.89 under \(P_r\); GPT-5.1 achieves SVR = 0.00 on standard pop-ups across all message types under both prompts
• Domain redirection: SVR = 0.00 for all agents under both prompts — agents never interact with the redirected link (but also rarely complete the task: TC ≈ 0.00)
• Banners: low SVR across models (≤ 0.07 under \(P_r\)), suggesting peripheral placement limits interaction
• Perception ablation (GPT-5.1, \(P_r\)): SoM achieves highest TC (0.63 on pop-up/banner vs. 0.21 for image-only), but SVR for price drift remains 1.00 across all perception settings — stronger perception does not improve safety

Limitations

• Evaluation is confined to a single e-commerce platform (VisualWebArena / OneStopShop); generalizability to other web domains is unverified
• Only 45 task instances from 9 seed templates; task diversity is intentionally constrained to isolate safety behavior, which may underestimate real-world complexity
• Seven deceptive patterns are representative but not exhaustive of real-world dark patterns
• The \(P_r\) safety prompt enumerates specific deceptive types, representing an idealized upper bound on prompt-based defense; real deployments rarely have such targeted prompts
• No defense mechanism is proposed; the work is purely evaluative
• Failure analysis (Section 5.4) is described as ongoing at the point the paper text is truncated

Relevance to Agentic AI / LLM Agents

This work directly stress-tests the real-world deployment readiness of autonomous web agents, revealing that the agentic action loop — where agents take irreversible financial actions (purchases, cart commits) — creates attack surfaces qualitatively different from those in conversational LLMs. The finding that SVR = 1.00 on cart add-ons and near-universal failure on price drift, even under explicit rule-based prompts, demonstrates that behavioral safety in agentic settings cannot be solved by prompt engineering alone, motivating research into perception-level anomaly detection or action-verification modules. The controlled injection methodology (WebDecept) is reusable and extensible, providing a reproducible testbed that the community can use to evaluate mitigations — closing a gap between adversarial-injection benchmarks and the dark-pattern threats agents will face in production e-commerce environments.