Skip to content

Same Payload, Different Channel: Measuring Trust Asymmetry in Tool-Using Language Models

🕒 Published (v1): 2026-05-30 06:38 UTC · Source: Arxiv · Venue: EMNLP 2026 · link

Why this paper was selected

EMNLP; trust asymmetry when tool outputs vs user instructions attack agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper introduces the Safety Asymmetry Score (SAS), a metric that isolates how a model's susceptibility to adversarial content shifts depending on whether that content arrives via user messages, tool descriptions, or tool outputs — keeping the malicious payload byte-for-byte identical across channels. Evaluated on 6 production LLMs, agent-native models are substantially more vulnerable through tool descriptions (+24.8 pp average SAS) while general models show the reverse. A mechanistic study on Llama 3.3 70B reveals the safety-relevant signal is causally present at mid-to-late residual stream layers (48 and 64 of 80) but encoded non-linearly.

Problem

Prior agent-safety benchmarks measure absolute vulnerability to attacks delivered via tool channels but do not measure whether the same malicious instruction succeeds at different rates depending on its delivery channel. This channel-dependent trust calibration — whether a model treats tool metadata versus user messages as more authoritative — is entirely uncharacterized. Existing defenses developed for indirect prompt injection (IPI) may therefore not transfer to the structurally distinct threat of tool poisoning, and vice versa.

Method

The core methodological contribution is the matched-payload design: for every test case \(c\), two prompts \(\langle c_\text{chat}, c_\text{tool} \rangle\) share byte-identical malicious instruction text (Malicious Action + Plausible Justification + user task + sampling parameters) and differ only in syntactic delivery channel. The Safety Asymmetry Score is defined as:

\[\text{SAS}(M; C) = \text{ASR}_\text{tool}(M) - \text{ASR}_\text{chat}(M), \quad \text{ASR}_x(M) = \frac{s_x(M)}{n_x(M)}\]

Three attack families are evaluated across 98 cases: tool poisoning (TP, 50 cases — malicious instructions embedded in tool description metadata, stratified into P1/P2/P3 subtypes), indirect prompt injection via tool output (IPI, 24 cases — malicious content injected into tool return values), and cross-tool shadowing (CTS, 24 cases — one tool's description piggybacks instructions targeting another). Scoring uses a six-class scheme (SUCCESS, IGNORED, REFUSED, DIRECT-EXECUTION, AMBIGUOUS, ERRORED) with a deterministic tool-call checker for the tool channel and an LLM judge for chat-mode non-refusals (inter-rater \(\kappa = 0.722\)). For mechanistic analysis, \(\ell_2\)-regularized logistic regression probes are fit on last-token residual-stream activations at layers \(\{16, 32, 48, 64\}\) of Llama 3.3 70B (80 layers total), and causal activation patching tests necessity and sufficiency via cosine shift scores in both forward (\(\text{adv} \to \text{ben}\)) and reverse (\(\text{ben} \to \text{adv}\)) directions.

Key Contributions

  • Safety Asymmetry Score (SAS): first metric isolating channel-as-sole-variable from payload, enabling cross-model comparison of trust calibration rather than absolute attack susceptibility.
  • Matched-payload benchmark: 98 cases across 3 attack families with per-family validator enforcing byte-level payload matching at case-generation time.
  • Behavioral finding: agent-native models (Nemotron, GPT-OSS 120B, Kimi K2.5) show mean SAS \(+24.8\) pp; general models average \(-5.6\) pp; group \(\Delta\)SAS \(= +30.4\) pp, driven almost entirely by tool poisoning.
  • Channel-trust decomposition: tool descriptions are treated as trusted instructions (positive SAS under TP) while tool outputs are treated as data (universally negative SAS under IPI), yielding a within-model trust hierarchy.
  • Mechanistic result: causal activation patching on Llama 3.3 70B identifies layers 48 and 64 as both necessary and sufficient for the safety-relevant representation, while linear probes fail — demonstrating non-linear encoding.
  • External validation: per-model TP ASR rankings replicate against MCPTox (Spearman \(\rho = 0.54\)) on 379 independently-constructed cases.

Results

  • Agent-native group mean SAS: \(+24.8\) pp; general group mean SAS: \(-5.6\) pp; \(\Delta\text{SAS} = +30.4\) pp (group-level, not population-confirmed at \(n=3\) each).
  • Tool-poisoning ASRs for agent-native models: 40–68% (tool channel) vs. 11–22% (chat); Nemotron 3 Super 120B peaks at 68% tool-TP ASR, 22% chat-TP ASR (\(\text{SAS}_\text{TP} = +46\) pp, CI [28, 64]).
  • IPI universally negative SAS across all 6 models: tool-channel ASRs 0–21%, chat-mode ASRs 8–50%.
  • CTS mixed; Llama 3.3 70B extreme outlier at \(\text{SAS}_\text{CTS} = -50\) pp (CI [\(-71\), \(-29\)]), driven by 95.8% chat-mode compliance with verbal piggyback instructions.
  • MCPTox replication: absolute ASRs differ substantially (ours 40–68% vs. MCPTox 3.7–18.8%, attributable to case construction differences), but rank correlation \(\rho = 0.54\) with Nemotron and GPT-OSS in top-2 positions on both sets.
  • Activation patching (Llama 3.3 70B): layers 16 and 32 shift scores near zero and CIs include zero; layers 48 and 64 show symmetric positive shifts in both directions (e.g., layer 64 forward: \(+0.095\) [+0.03, +0.17], reverse: \(+0.090\) [+0.03, +0.16]).
  • Cross-channel linear probe transfer: collapses to near-chance (chat→tool = 0.50, tool→chat = 0.55 at layer 64), confirming the in-channel separating direction is largely channel-specific.

Limitations

  • Small model roster: \(n = 3\) per category makes group-level SAS an effect-size estimate, not a statistically confirmed population claim; inferential statistics are underpowered.
  • Affordance confound: matched-payload construction holds content identical but not action affordances — tool-channel cases register actual tools; chat-mode cases do not. SAS therefore conflates channel trust with affordance presence; the authors argue the trust component dominates (IPI inversion, chat-mode SUCCESS scoring) but cannot fully disentangle.
  • Mechanistic scope: Llama 3.3 70B is the only model with accessible internals; cross-model replication of patching results is left as future work. IPI and CTS families excluded from mechanistic analysis.
  • Single-layer patches only: remote tracing API limits cumulative multi-layer interventions; causal claims are therefore layer-local.
  • MCP simulation: MCP is simulated at content layer (no real MCP server); transport-layer threats (rebinding, MITM) are out of scope.
  • Scoring asymmetry: tool-channel SUCCESS is deterministic; chat-mode SUCCESS relies on an LLM judge for non-refused traces, introducing judge variability on one arm of SAS (bounded at \(< 5\) pp by robustness check).
  • 98 cases total: relatively small benchmark; per-family cell sizes (24–50 cases) limit bootstrap CI width.

Relevance to Agentic AI / LLM Agents

This work directly addresses one of the most underexplored attack surfaces in LLM agent deployments: the implicit trust hierarchy a model assigns to different input channels in tool-use pipelines. The finding that agent-native models treat tool description metadata as authoritative instructions — more so than the user's own message — is a significant security signal for anyone deploying MCP-based agents or multi-tool orchestration systems. The channel-trust decomposition (descriptions trusted, outputs treated as data) provides a mechanistic framing for why defenses designed for classical indirect prompt injection may fail against tool poisoning, directly relevant to hardening agent tool-call boundaries. The causal patching result opens a concrete direction for activation-level safety monitoring: the safety-relevant signal exists at layers 48–64 in a 70B model but requires non-linear probing methods, suggesting that linear activation monitors deployed in agentic pipelines will miss it.