Skip to content

Breaking and Fixing Defenses Against Control Flow Hijacking in Multi-Agent Systems

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Vitaly Shmatikov (Cornell); attacks and fixes for control-flow hijacking in MAS

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Existing alignment-check defenses (e.g., LlamaFirewall) for multi-agent systems can be evaded by control-flow hijacking (CFH) attacks that camouflage malicious instructions as environmental errors routed through trusted agents. The authors introduce CONTROLVALVE, which enforces a pre-generated control-flow graph (CFG) and per-edge contextual rules at the orchestration layer, achieving 0% attack success rate across all evaluated CFH scenarios while preserving benign task functionality.

Problem

Multi-agent systems (MAS) delegate sub-tasks to black-box agents that ingest untrusted external content (emails, web pages, files). Control-flow hijacking exploits this delegation via confused-deputy attacks: adversarial content masquerades as legitimate error messages with step-by-step "fix" instructions that re-route orchestrator planning toward unsafe agents. Alignment-check defenses (LlamaFirewall) that ask whether an action is "related to" and "likely to further" the user's goal are fundamentally brittle because CFH attacks frame unsafe actions as necessary for task completion, and because no single system vantage point has complete context visibility.

Method

Attack analysis: The authors craft CFH-Hard attacks that evade alignment checks by (a) presenting unsafe payloads as the only path to fix an environmental error, (b) routing them through an already-trusted agent (confused deputy), and (c) framing the malicious action as contextually plausible rather than unconditionally harmful. These evade LlamaFirewall backed by Llama, GPT-4o, GPT-4o-mini, and o4-mini.

CONTROLVALVE defense: 1. Planning stage (pre-execution, before untrusted content is ingested): Given the user's task and available agents, an LLM generates a task-specific context-free grammar over agent-call tokens, compiled via Lark into a parser that defines the language of admissible agent-invocation traces (permitted CFG). For each directed edge \(A \to B\) in the CFG, an LLM also generates \(\leq 3\) natural-language contextual rules enforcing input validation, contextual appropriateness, and data provenance. 2. Execution stage: Before each agent transition \(A \to B\), an LLM judge checks (i) whether the transition is a valid edge in the CFG parser, and (ii) whether the edge-specific contextual rules are satisfied. The judge is not asked to evaluate the merits or justifications of the rules—only binary compliance. On failure, CONTROLVALVE attempts up to 3 re-planning steps; otherwise it blocks and reports to the user.

The CFG generation uses few-shot examples of parseable grammars (not attack demonstrations). Edge-specific rules are zero-shot. The defense operates only at the orchestration layer (inter-agent messages), requiring no access to individual agents' internal states or tool calls.

Key Contributions

  • Demonstration that CFH attacks with error-masquerading and confused-deputy routing systematically evade alignment-check defenses (LlamaFirewall across 4 LLMs), improving attack success rate up to \(12\times\) over prior CFH exploits on 8 of 12 attack configurations
  • CFH-Hard: a new dataset of 16 tasks × 14 attacks covering coding (reverse shell) and computer-use (email exfiltration via cc:) MAS scenarios, plus a MAS wrapper for AgentDojo's Slack environment
  • First empirical demonstration that vague, non-adversarial inputs cause accidental control-flow violations (unauthorized data access/exfiltration) in undefended MAS
  • CONTROLVALVE: a task-agnostic, zero-shot CFI-inspired defense that (1) pre-generates CFGs and per-edge contextual rules, (2) enforces them via a narrow LLM judge at runtime, and (3) achieves 0% attack success rate across all evaluated scenarios

Results

  • CONTROLVALVE vs. CFH-Hard (coding, reverse shell): 0% attack success rate (ASR) across all 5 CFH-Hard templates (Generic, Python, WordPress, File-Not-Found, Agent-Not-Found); undefended baseline: 97–100%; best LlamaFirewall variant (o4-mini): 10–57%
  • CONTROLVALVE vs. CFH-Hard (computer use, email cc: exfiltration): 0% ASR across all 5 CFH templates; undefended baseline: 67–89%; best LlamaFirewall variant: 6–72%
  • Accidental violations (computer use): CONTROLVALVE reduces accidental cc: exfiltration to 0%; undefended MAS: 89%; best LlamaFirewall: 50%
  • CONTROLVALVE vs. conventional IPIs: Blocks all tested IPI formats (InjecAgent, Greshake, AgentDojo); LlamaFirewall allows up to 63% ASR on computer-use AgentDojo presentation
  • Least privilege: Effective only on 2 of 10 coding tasks (tasks not requiring the attacker's target agent); fails on all computer-use tasks because attacks exploit agents the user legitimately needs
  • Azure Content Filters: 0% IPI/CFH blocked; only triggered on benign CONTROLVALVE checks

Limitations

  • CFG and edge rules are LLM-generated and may be incorrect, too permissive (over-approximation), or too restrictive, causing defense failures or blocking legitimate executions
  • An over-approximate CFG may permit "compliant CFH" attacks (analogous to CFI evasion via JOP/ROP in systems security)—this remains an open question for MAS
  • Too-restrictive CFGs degrade MAS functionality by preventing legitimate adaptive re-planning in response to genuine runtime errors
  • Runtime overhead is comparable to LlamaFirewall but may be intractable for latency-sensitive deployments
  • Evaluation limited to Magentic-One/AutoGen configuration; transfer to other platforms is argued but not empirically verified

Relevance to Agentic AI / LLM Agents

This paper directly addresses a critical attack surface of deployed multi-agent systems: the orchestration layer itself, which is exposed to adversarial manipulation through the very delegation mechanism that makes MAS powerful. It formalizes the safety–functionality tension that makes semantic alignment checks insufficient and proposes a structural (control-flow integrity) alternative that is both more rigorous and harder to reason-through. For researchers building or deploying LLM agent pipelines, CONTROLVALVE's CFG-enforcement paradigm offers a principled complement to prompt-level defenses, particularly relevant as MAS deployments expand under protocols like Google's Agent2Agent. The discovery of accidental (non-adversarial) control-flow violations also highlights a previously underexplored failure mode for real-world agentic systems.