Skip to content

Agents That Know Too Much: A Data-Centric Survey of Privacy in LLM Agents

🕒 Published (v1): 2026-06-25 05:44 UTC · Source: Arxiv · link

Why this paper was selected

Data-centric survey of privacy across all LLM agent components (memory, APIs, actions); practical threat taxonomy for builders

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This survey organizes LLM agent privacy around the data an agent touches rather than by attack type, introducing the concept of a "data agent" as an LLM agent that queries, retrieves, transforms, and acts on structured data. It taxonomizes six data surfaces, their associated privacy risks, and governance mechanisms, then identifies the critical gap: no benchmark evaluates an agent end-to-end under a single explicit privacy policy.

Problem

Prior agent-security surveys organize by attack type or architectural component, leaving the combined account of an end-to-end data agent fragmented across communities (NL-to-database, RAG, agent memory, prompt injection, access control). The shift from stateless chatbots to multi-step, stateful, tool-using agents eliminates the single output-level enforcement point: sensitive data can now leak through generated queries, retrieved rows, tool call arguments, reasoning traces, memory writes, and inter-agent messages—none of which output filters address. No benchmark covers all these surfaces under one coherent privacy policy.

Method

The survey adopts a data-centric taxonomy structured across three layers—data surfaces, privacy risks, and governance mechanisms—linked by three cross-reference tables:

  • Six data surfaces (§3): databases/warehouses, tabular/file data, RAG corpora/vector stores, tools/external APIs, agent memory, and multi-agent communication channels. Each surface is characterized by its sensitive unit and leakage pathway.
  • Privacy risks (§4) decomposed into outcomes (what is disclosed: direct output, intermediate-step, schema/metadata, inference/aggregation, compositional, policy/consent, cross-session) and vectors (how triggered: prompt injection, memory poisoning, over-permissioned access, compromised peer), connected via Table 4.
  • Governance mechanisms (§5): access control, information-flow control (IFC), runtime governance, privacy-preserving transformations, differential privacy, confidential retrieval, contextual integrity, auditing/provenance, and human oversight. Mechanism-to-outcome coverage is mapped in Table 5.
  • Benchmark audit (§6) identifies what each existing benchmark covers and pinpoints the missing instrument: an end-to-end workflow benchmark that drives an agent across all surfaces under one explicit privacy policy.

Key Contributions

  • Motivates the data agent framing: distinguishes privacy properties of stateful, tool-using data agents from those of stateless chatbots and generic tool-using agents.
  • Three interconnected taxonomies (surfaces → outcomes → controls) with cross-tables intended as a reusable reference for builders and reviewers.
  • Identifies information-flow control as the sole mechanism covering both compositional leakage and cross-session inference leakage—the two least-protected risk classes.
  • Surveys and compares existing benchmarks, characterizing each by which surfaces and risk outcomes it covers.
  • Enumerates open problems, including the absence of end-to-end evaluation, memory deletion/revocation, and utility-first privacy for multi-step queries.
  • Connects two communities—agent security and data management—that have addressed agent privacy in isolation; specifically imports the text-to-SQL and enterprise data governance literature.

Results

This is a survey paper; there are no empirical results. Qualitative findings reported from surveyed works: - Existing NL-to-database benchmarks show models return withheld data when prompted adversarially [cited as ref 97]. - A survey of data-science agent systems found the great majority released without explicit trust or safety mechanisms [ref 89]. - A full-stack multi-agent benchmark finds internal channels raise total system exposure beyond what any per-component evaluation detects [ref 29/47 implied]. - Only IFC, among all surveyed governance mechanisms, covers both compositional and cross-session inference leakage. - No current benchmark drives an agent across all data surfaces under a single explicit privacy policy.

Limitations

  • Survey scope is 2023–2026; preprint results are treated as provisional where not independently reproduced.
  • Coverage of training-data memorization in standalone models is explicitly excluded except where it intersects agent data surfaces.
  • The utility-privacy trade-off is characterized qualitatively; no formal bounds on the cost of governance mechanisms are derived.
  • The claim that IFC is uniquely sufficient for compositional and cross-session leakage rests on the coverage mapping (Table 5) rather than empirical evaluation.
  • "Data agent" as a durable paradigm vs. transient framing is acknowledged as contested.

Relevance to Agentic AI / LLM Agents

This survey is directly foundational for anyone building or evaluating LLM agents that operate over real data: it makes explicit that multi-step, stateful, tool-using agents have fundamentally different privacy threat surfaces than chatbots, and maps which governance primitives address which threat class. The identification of information-flow control as the uniquely capable mechanism for compositional and cross-session leakage gives agent designers a concrete architectural recommendation. The benchmark gap finding—no end-to-end privacy evaluation across all data surfaces—sets a clear research agenda for the agentic AI community. It also bridges agent security with the data management literature (text-to-SQL, differential privacy, contextual integrity), which is underrepresented in mainstream agent research.