Skip to content

Agent Data Injection Attacks are Realistic Threats to AI Agents

🕒 Published (v1): 2026-07-06 14:07 UTC · Source: Arxiv · link

Why this paper was selected

Data injection attacks beyond prompt injection; Luyi Xing (UIUC); realistic threat coverage for agent pipelines

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper introduces Agent Data Injection (ADI), a new class of indirect prompt injection attack that causes LLMs to misinterpret attacker-controlled untrusted data as trusted metadata (e.g., sender identities, UI element IDs, tool response formats) rather than as instructions. The core mechanism, probabilistic delimiter injection, exploits the fact that LLMs interpret data structure probabilistically and thus misread even escaped/inexact format delimiters as structural boundaries. Real-world attacks achieve arbitrary click execution on web agents and remote code execution on coding agents, while bypassing existing defenses.

Problem

Prior AI agent security research focused almost exclusively on instruction injection—where untrusted data is misread as an instruction—and defenses address only the instruction/data boundary. A deeper, unexplored vulnerability exists within agent data itself: agents do not isolate trusted data (tool-generated metadata such as sender fields, element identifiers, tool call structures) from untrusted data (user-generated content such as email bodies or webpage text). This allows an attacker to forge trusted metadata without ever injecting an instruction, evading all existing instruction-injection mitigations.

Method

The attack formalizes agent data as \(D = (D_T, D_U)\), where \(D_T\) is trusted (tool/agent-generated) and \(D_U\) is untrusted (externally controlled). Injection of attacker payload \(D_A\) into \(D_U\) achieves ADI when:

\[\text{LLM}(I,\,(D_T,\,D_U \| D_A)) \approx_{\text{ADI}} \text{LLM}(I,\,(D_T \| D_A,\,D_U))\]

The mechanism is probabilistic delimiter injection: the attacker embeds character sequences that mimic the structural delimiters of the agent data format (JSON {}, newline-based element identifier entries, XML tags, etc.) inside an untrusted field. Because LLMs interpret data structure probabilistically—not via a deterministic parser—even escaped or slightly malformed delimiters (e.g., \" instead of ") are probabilistically misread as valid structural boundaries, causing the LLM's view of the data tree to diverge from the tool's actual output. Three concrete attack variants are demonstrated:

  1. Arbitrary click attack (web agents): A product review injects a fake button "Read More" [ref_3] entry mimicking the webpage summary format; the LLM resolves the spoofed identifier to the legitimate "Buy Now" DOM element and clicks it. Tested on Claude in Chrome, Antigravity, Nanobrowser.
  2. Remote code execution (coding agents): A GitHub issue comment injects JSON delimiters to spoof the author metadata field as a maintainer, tricking the agent into executing maintainer-trusted commands. Tested on Claude Code, Codex, Gemini CLI.
  3. Supply chain attack (coding agents): A malicious pull request injects a fake tool response claiming code review passed, causing the agent to merge without inspecting actual diffs.

Key Contributions

  • Identifies the trusted/untrusted data boundary within agent context as a previously unexploited attack surface distinct from the instruction/data boundary.
  • Introduces probabilistic delimiter injection as the first technique that systematically exploits LLM probabilistic misinterpretation of inexact (parser-invalid) delimiters—contrasting with classical deterministic delimiter injection (SQL injection, XSS) that requires exact syntactic matches.
  • Demonstrates proof-of-concept ADI attacks against six real-world production agents across two categories (web agents and coding agents) spanning three attack impact types.
  • Evaluates ADI against existing IPI defenses (model hardening, input guardrails, dual-LLM), showing most fail to prevent ADI.
  • Releases a probabilistic delimiter injection benchmark and an extended AgentDojo benchmark with ADI attack scenarios.
  • Conducted responsible disclosure to Anthropic, OpenAI, Google, and Nanobrowser; OpenAI, Google, and Anthropic acknowledged the issues.

Results

  • Standalone LLM vulnerability (probabilistic delimiter injection benchmark): Attack Success Rate (ASR) of 31.3%–43.3% on JSON-formatted data and 33.3%–100.0% on web DOM data across six off-the-shelf models.
  • Against existing IPI defenses: State-of-the-art agent defenses reduced instruction injection ASR to 0.0%–0.7%, but ADI achieved up to 50.0% ASR against the same defenses—demonstrating a fundamental bypass.
  • Real-world agents confirmed vulnerable: Arbitrary click attacks confirmed on Claude in Chrome, Antigravity, and Nanobrowser; RCE and supply-chain attacks confirmed on Claude Code, Codex, and Gemini CLI.

Limitations

  • The attacker is assumed to know the agent's data format; while the paper argues this is recoverable in practice, it is still an assumption that narrows the threat model.
  • The paper demonstrates attacks but does not propose or evaluate a concrete defense—analysis of existing mitigations shows they fail, but no novel mitigation is provided.
  • Evaluation of ASR is on a benchmark; end-to-end ASR in fully realistic, varied agent deployments with context-specific system prompts may differ.
  • The three attack types cover web and coding agents; generalizability to other agent categories (e.g., embodied, database, enterprise workflow agents) is shown only in an appendix and not fully characterized in the main evaluation.
  • The supply-chain and RCE attacks require the agent to be operating in an automated/agentic workflow with elevated trust placed on tool responses; human-in-the-loop deployments would reduce impact.

Relevance to Agentic AI / LLM Agents

ADI reveals that the security model underlying nearly all current LLM agent architectures is fundamentally flawed: agents conflate trusted tool-generated metadata with untrusted external content in a single LLM context, with no enforced isolation boundary. This is directly analogous to the classic confused-deputy and data/code separation problems in systems security, now manifesting in the agentic layer. For researchers tracking agentic AI, this establishes a new attack taxonomy that goes beyond instruction injection and demands a fine-grained trust model inside the agent context—a design challenge for any framework (LangChain, Claude tool use, OpenAI function calling, Gemini CLI) that serializes mixed-trust data into a single prompt. The results also signal that benchmark-level defenses (guardrails, dual-LLM) developed against instruction injection will not transfer to this new threat class without architectural changes.