Searching for Privacy Risks in LLM Agents via Simulation¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Diyi Yang (Stanford); first systematic study of privacy extraction risks in LLM agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper presents a search-based framework that automatically discovers privacy attack and defense strategies for multi-agent LLM systems via large-scale simulation. By alternating between LLM-optimized attack and defense instruction search, the framework surfaces sophisticated threats—such as consent forgery and multi-turn impersonation—that simple enumeration or manual design would miss. The discovered strategies generalize across backbone models and privacy scenarios.

Problem¶

Prior agent-privacy research focuses on benign settings (under-specified user instructions or static malicious environment elements), failing to model the dynamic, adaptive threat posed by malicious agents that actively engage in multi-turn dialogues to extract sensitive information. No systematic methodology existed for discovering such long-tail, interaction-driven vulnerabilities before they manifest in deployment.

Method¶

The framework models each privacy norm (sourced from PrivacyLens, grounded in contextual integrity theory) as a three-agent simulation: a data subject who holds sensitive information, a data sender (defender), and a data recipient (attacker). Agents are implemented with the ReAct architecture over four mock applications (Gmail, Facebook Messenger, Notion, email).

Alternating search co-evolves attack and defense instructions: - Attack search uses an LLM optimizer $F$ that reflects on simulation trajectories $\{(a^r, E^r) \mid 1 \le r \le k\}$ to propose better attacker instructions $a^{k+1}$. To escape local optima, $N=30$ parallel threads are initialized with distinct instructions and search independently. - Cross-thread propagation: when a new best instruction (re-evaluated with $P=10$ simulations) is found, its top trajectories are shared to all threads: $E^k \leftarrow \text{Select}(\bigcup_{i=1}^{N} S_i^k)$. - Defense search runs single-threaded across all $Q$ scenarios simultaneously, minimizing average leak velocity over $K=10$ steps.

The primary metric is leak velocity: $$s = \frac{1}{K}\sum_{i=1}^{K}\left(1 - \frac{\log l_i}{\log l_i + 1}\right)$$ where $l_i \in [1, +\infty)$ is the action step at which the $i$-th sensitive item is leaked ($s=0$ if no leak; $s=1$ if all items leak on the first action).

Key Contributions¶

A simulation-based adversarial search framework for agent–agent privacy threats, framing attacker/defender instructions as optimizable objects.
Parallel search with cross-thread propagation for efficient exploration of rare, long-tail attack strategies.
Empirical discovery of an attack evolution trajectory: direct request → fake urgency/invented authority → impersonation + consent forgery + multi-turn coordination.
Discovery of a corresponding defense evolution: no constraint → rule-based consent-required → identity-verifying state machine with anti-spoofing.
Demonstration of cross-model and cross-scenario transferability of discovered strategies.

Results¶

Baseline (basic instructions, gpt4.1-mini vs. gpt4.1-mini): 31.2% leak velocity (LV), 37.6% leak rate (LR)—nontrivial leakage even with privacy-augmented defender instructions.
Model scaling for attack: upgrading attacker backbone from gpt4.1-mini → gpt4.1 yields marginal gain (31.2% → 33.0% LV), suggesting better models alone don't produce better attacks.
Model scaling for defense: upgrading defender backbone yields substantial improvement (31.2% → 16.5% LV), indicating defense benefits more from model capability.
After alternating search (Training-5, gpt4.1-mini backbone): the framework finds $A_1, D_1, A_2, D_2$; further search for $A_3$ fails to improve leakage, suggesting the discovered $D_2$ (state-machine with identity check) is highly robust.
Cross-model transfer (Table 2): discovered attacks and defenses transfer across gpt4.1, gpt4.1-nano, gemini-flash, qwen3-32B, and gptoss-20B with largely consistent relative rankings.
Evaluation reliability: automated leakage detection (gemini-flash) achieves 98.5% agreement with human annotators over 200 sampled actions.

Limitations¶

Defense search is restricted to prompt-based instructions; more complex programmatic protocols (e.g., firewalls, extra privacy agents) are not explored, only noted as future extensions.
Training set is only $Q=5$ scenarios to reduce compute; generalization to the full diversity of real-world privacy norms is assumed but not exhaustively verified.
Backbone agents use ReAct; other agent architectures may exhibit different vulnerability profiles.
The simulation environment (four mock apps) may not capture all transmission channels or realistic UI surfaces present in deployed agents.
The framework does not address attacks that operate across session boundaries or through side channels outside the simulated application layer.

Relevance to Agentic AI / LLM Agents¶

This work directly addresses a critical but under-studied threat surface in multi-agent systems: agent-to-agent social engineering, where one LLM agent actively manipulates another across multiple turns to exfiltrate private data. For researchers tracking agentic AI, it provides both a concrete threat taxonomy (impersonation, consent forgery, fake urgency) and a reusable simulation-plus-search methodology for red-teaming agent pipelines before deployment. The finding that better backbone models improve defense far more than attack has direct implications for safe multi-agent system design. The state-machine defense pattern ($D_2$) offers a transferable, prompt-implementable blueprint for privacy-preserving agent instructions.