Skip to content

ChatInject: Abusing Chat Templates for Prompt Injection in LLM Agents

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Chat template abuse enables prompt injection bypassing instruction hierarchy

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ChatInject is an indirect prompt injection attack that forges chat template tokens (e.g., <|im_start|>system) inside tool outputs to make LLM agents treat attacker-controlled content as higher-priority instructions. A multi-turn variant embeds a synthetic persuasive dialogue within a single injection payload. On InjecAgent, it lifts average ASR from 15.13% to 45.90% (multi-turn: 52.33%); on AgentDojo, from 5.18% to 32.05%.

Problem

Existing indirect prompt injection techniques operate on plain text and fail to exploit two structural properties of modern LLM agents: (1) role-based instruction hierarchies enforced through special chat-template tokens, and (2) the model's learned dependency on multi-turn conversational context. Prompt-based defenses trained against plain-text injection remain largely untested against template-formatted payloads.

Method

Payload construction: Given a malicious instruction \(I_a\) or attacker-crafted conversation history \(C_a = \{(r_i^a, m_i^a)\}_{i=1}^n\), the attack applies a template function \(T_{\text{type}}\) that wraps roles in the target model's native special tokens (e.g., <|im_end|>, <|im_start|>system), producing four payload variants: - \(T_{\text{plain}}(I_a)\) — Default InjecPrompt (baseline plain text) - \(T_{\text{model}}(I_a)\) — InjecPrompt + ChatInject (instruction wrapped in system/user role tags) - \(T_{\text{plain}}(C_a)\) — Default Multi-turn (plain-text simulated dialogue) - \(T_{\text{model}}(C_a)\) — Multi-turn + ChatInject (GPT-4.1-synthesized 7-turn dialogue wrapped in role tags)

Agentic hooks: Two extensions inject <think> tokens (reasoning hook, appending "Sure!" to steer internal reasoning) and <tool> tokens (tool-calling hook, scaffolding a forged tool-call with the target tool name).

Cross-model transfer: Template cosine similarity (measured via lighter-weight same-family proxies) predicts transferability; a mixture-of-templates ensemble covers unknown closed-source targets.

Key Contributions

  • Identification of chat-template token forging as a new attack surface for indirect prompt injection, distinct from prior plain-text and jailbreak approaches.
  • ChatInject and its Multi-turn variant: concrete payload generation pipeline using GPT-4.1-synthesized persuasive dialogues wrapped in model-specific role tokens.
  • Agentic extensions exploiting <think> (reasoning) and <tool> (tool-calling) template hooks.
  • Empirical demonstration that template cosine similarity predicts cross-model attack transferability, including against closed-source models with unknown templates.
  • Evaluation showing existing prompt-based defenses are ineffective, especially against multi-turn variants.

Results

  • InjecAgent ASR: Default InjecPrompt average 15.13% → ChatInject 45.90% → Multi-turn + ChatInject 52.33%.
  • AgentDojo ASR: Default InjecPrompt average 5.18% → ChatInject 32.05%.
  • Per-model highlights (InjecAgent ASR): GLM-4.5: 0.0% → 57.3% (ChatInject) → 71.5% (multi-turn); Llama-4: 50.1% → 79.4% → 88.3%; Qwen-3: 8.5% → 39.4% → 65.9%; Grok-2: 16.5% → 17.7% (minimal gain, weak role delimiters).
  • Utility degradation (AgentDojo): ChatInject consistently reduces task-completion utility; e.g., Qwen-3 utility drops from 50.9% to 28.3% under ChatInject.
  • Tool-calling hook: Further increases ASR even when tools are not named explicitly (AgentDojo setting).
  • Cross-model transfer (AgentDojo): Foreign OS templates frequently exceed Default InjecPrompt ASR; closed-source models (GPT-4o, Grok-3, Gemini-2.5-Pro) show high transferability with OS proxy templates.
  • Template similarity correlation: Higher cosine similarity between injected and target template → higher ASR and lower utility, with Qwen-3 template transferring most broadly (avg 21.4% InjecAgent, 16.8% AgentDojo).

Limitations

  • Attack requires knowledge of (or ability to guess) the target model's chat template tokens; effectiveness is reduced for models with weak delimiters (Grok-2) or unknown templates, though mixture-of-templates partially mitigates this.
  • Cross-model template similarity is estimated via lighter-weight same-family proxy models due to resource constraints, introducing approximation error.
  • Multi-turn dialogue generation depends on GPT-4.1 and manual review; scalability to diverse attack targets is constrained.
  • Defenses are evaluated only for existing prompt-based methods; adversarial training or structural input sanitization countermeasures are not tested.
  • Evaluation is restricted to AgentDojo and InjecAgent; generalization to other agentic pipelines (e.g., RAG-heavy or code-execution agents) is unverified.

Relevance to Agentic AI / LLM Agents

ChatInject directly targets the tool-use loop that defines agentic systems: the attack surface is the gap between tool output (low-priority data) and the instruction hierarchy (high-priority roles), which is exploited by forging the structural tokens agents rely on to segment context. This is a qualitatively different threat model from plain-text injection because it abuses the instruction-following machinery that makes LLMs capable agents in the first place — the same role-token conventions used to enforce safe behavior become vectors for authority escalation. The finding that <think> and <tool> hooks further amplify attacks signals that richer agentic scaffolding (chain-of-thought, ReAct-style tool calls) enlarges the attack surface. For practitioners deploying LLM agents that process untrusted external data, this work establishes that role-hierarchy defenses are insufficient without structural sanitization of template tokens in tool responses.