Skip to content

ToolPrivacyBench: Benchmarking Purpose-Bound Privacy in Tool-Using LLM Agents

🕒 Published (v1): 2026-06-26 13:08 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ToolPrivacyBench audits whether tool-using LLM agents route current-task private atoms only to purpose-authorized tools across multi-step workflow trajectories. On 2,150 cases covering nine agents, high task completion (TaskSuccess 92.23–97.70) coexists with substantial privacy over-disclosure (MT-POI 19.19–28.04), especially through tickets, handoffs, and free-text business fields.

Problem

Existing function-calling benchmarks evaluate task completion and API correctness; privacy benchmarks focus on final-response leakage or privacy judgments. Neither captures purpose-bound information flow across an executed multi-tool trajectory — a private field (e.g., a medical symptom) may be necessary for one tool (clinical record) but unauthorized for another (billing, notification, handoff). No prior benchmark systematically evaluates whether agents respect need-to-know disclosure boundaries during tool invocation.

Method

Each case is a quadruple \(c = (x, \mathcal{T}, \mathcal{P}, \mathbf{A})\): user task \(x\), tool set \(\mathcal{T}\), private-atom set \(\mathcal{P}\), and field-tool authorization matrix \(\mathbf{A} \in \{0,1\}^{|\mathcal{P}|\times|\mathcal{T}|}\). The pipeline: (1) offline dataset construction — 1,150 fully synthetic privacy-sensitive business workflows + 1,000 cases adapted from existing multi-tool/function-calling benchmarks, spanning healthcare, finance, insurance, tax, recruiting, education, IT operations, and software security; (2) agent execution via the OpenClaw stack against mock business backends, producing a trajectory \(\tau = [(t_1, a_1), \ldots, (t_K, a_K)]\); (3) post-execution audit — a disclosure detector \(D(p_j, a_k)\) checks whether each private atom appears in each tool argument, and an authorization reasoner compares against \(\mathbf{A}\) to flag over-disclosure \(O_{j,k} = D(p_j, a_k) \cdot \mathbb{I}[A_{j,i(k)} = 0]\). Three metrics are reported: TaskSuccess (task completion, workflow coverage, required-fact delivery), MT-POI (aggregated multi-tool privacy over-disclosure risk), and FOR (field opportunity leakage rate — fraction of unauthorized field-tool opportunities where a private field is actually disclosed). The policy knowledge base is used only for post-execution audit, not supplied to agents at inference time.

Key Contributions

  • Formalizes a previously unmeasured failure mode: current-task private atoms that are necessary for one workflow step get routed to tools, sinks, or free-text fields that do not require them.
  • Builds ToolPrivacyBench with 2,150 multi-tool cases across eight domains, each annotated with a purpose-bound policy knowledge base containing private atoms, tool purposes, sink types, authorization relations, and backend audit evidence.
  • Evaluates nine widely used LLM agents and shows that high task completion does not imply appropriate privacy disclosure — auditing the full tool-call trajectory is necessary.

Results

  • TaskSuccess ranges from 92.23 to 97.70 across agents on the synthetic private split — most models can complete multi-tool tasks effectively.
  • MT-POI ranges from 19.19 to 28.04, indicating persistent purpose-bound over-disclosure even when tasks succeed.
  • Highest-leakage sinks: tickets (FOR 51.43) and handoffs (FOR 34.79) — i.e., in over half of opportunities, agents leak unauthorized private atoms into ticket descriptions.
  • Free-text business fields (e.g., message, note, summary, work_notes) are repeatedly identified as over-disclosure channels.

Limitations

  • Non-adversarial scope only: benign user requests, no prompt injection, jailbreaks, or malicious intent — evaluates unintentional over-disclosure, not attacks.
  • Current-task focus: does not measure training-data extraction, long-term memory leakage, cross-task memory contamination, or standing-permission escalation.
  • Trusted-infrastructure assumption: tool wrappers, mock backends, audit logs, and the evaluator itself are assumed trustworthy — no tampering or man-in-the-middle.
  • The policy knowledge base is not provided to agents as a defense; the benchmark measures the problem, not a solution.

Relevance to Agentic AI / LLM Agents

This paper identifies a blind spot in agent evaluation: tool-call success is not privacy-correctness. For researchers building LLM agents that handle sensitive data (healthcare, finance, enterprise), the benchmark provides a concrete framework and metrics for auditing purpose-bound information flow. It connects to the broader theme of agent safety by showing that functional competence and privacy compliance are orthogonal — a finding that should inform runtime monitoring, tool-call filtering, and disclosure-aware agent architectures.