Adaptive Evaluation of Out-of-Band Defenses Against Prompt Injection in LLM Agents¶

🕒 Published (v1): 2026-06-25 00:35 UTC · Source: Arxiv · link

Why this paper was selected

Adaptive red-teaming evaluation framework for out-of-band prompt-injection defenses in tool agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Out-of-band (action-level) defenses against prompt injection in LLM agents — CaMeL, FIDES, Progent, RTBAS, FORGE — are validated only on static attack benchmarks, the same methodology that caused twelve in-band defenses to collapse under adaptive attacks. This paper organizes these defenses through classical security primitives (Biba integrity, reference monitors, least privilege) and runs an independent adaptive evaluation of Progent on an open-weight agent (Qwen2.5-7B), finding the deterministic gate held across three runs despite a hand-crafted defense-aware attack.

Problem¶

In-band defenses (classifiers, guardrail models, adversarial fine-tuning) were broken at >90% success by adaptive attackers who knew the defense. The emerging second generation of out-of-band action-level defenses reports near-zero attack success on static benchmarks — but those benchmarks use fixed attack sets chosen before the defense existed, not attacks optimized against the specific monitor. The field lacks a standardized adaptive evaluation protocol for this class of defense, leaving current confidence unverified.

Method¶

Two components:

Systematization: The authors map CaMeL, FIDES, Progent, RTBAS, Conseca, and FORGE onto classical security primitives — Biba integrity (no-read-down / no-write-up), Anderson's reference monitor (complete mediation, tamperproof, verifiable), Saltzer–Schroeder principles, and information-flow control with taint labels. Eight comparison dimensions are defined: enforcement primitive, deterministic vs. LLM gate, monitor placement, integrity/confidentiality coverage, implicit-flow handling, cost, and retrofit feasibility.
Adaptive evaluation of Progent: On AgentDojo (banking, slack, workspace suites; first 8 user tasks × all injection tasks), using Qwen2.5-7B-Instruct served via vLLM on a single H200 (self-hosted, no external API). Three conditions per suite: (A) undefended + standard attack, (B) Progent + standard attack, (C) Progent + adaptive attack. The adaptive attack is a defense-aware injection that disguises the malicious action as a benign pre-authorized step to manipulate Progent's policy-update mechanism. Three full runs are reported (mean ± std) to quantify variance under temperature-0 agent nondeterminism.

Key Contributions¶

Structured comparison of six out-of-band defenses through Biba/reference-monitor/IFC primitives across eight evaluation dimensions (Table 2)
Formal specification of a threat model and evaluation protocol for adaptive attacks against action-level defenses
Independent reproduction and extension of Progent's own adaptive-attack analysis (their Appendix E) on a weak open-weight model (Qwen2.5-7B vs. GPT-4o), with variance measured over three runs
Identification of four persistent gaps: missing adaptive evaluation methodology, underspecified provenance assignment, unresolved in-the-loop task security, and unmitigated text-to-text harms

Results¶

Undefended baseline: mean attack success rate 25.8% on the fixed subsample
Progent + standard attack: mean ASR 4.2% — roughly sixfold reduction
Progent + adaptive attack (defense-aware): mean ASR 2.6% — no degradation relative to standard attack; the deterministic gate held
Progent's original paper reports 39.9%→1.0% on full AgentDojo with GPT-4o; this reproduction uses Qwen2.5-7B and a smaller subsample, so numbers differ but directional result is consistent
White-box GCG optimization against the policy monitor was not attempted and remains an open threat

Limitations¶

Single small-scale evaluation with one black-box adaptive attack template; not a comprehensive adversarial assessment
Policy-authoring LLM substituted from GPT-4o to Qwen2.5-7B, which may weaken both the defense and the baseline
Travel suite excluded (not stated why in available text)
Only Progent tested; CaMeL, FIDES, FORGE not empirically evaluated
White-box (GCG) attacks against the deterministic monitor not run
Provenance assignment to trusted vs. untrusted sources is underspecified in all surveyed systems; if the user pastes attacker-controlled content, labels are wrong at the source
In-the-loop tasks (where untrusted data must drive a consequential action) have no principled security model
Text-to-text harms (misleading summaries, injected user instructions) are entirely outside the action-mediation threat model

Relevance to Agentic AI / LLM Agents¶

This paper directly addresses the authorization layer of tool-using agents — the enforcement boundary that decides which actions an agent may actually execute, independent of what the LLM was persuaded to request. The adaptive evaluation gap it identifies is critical for any production agent deployment: static benchmark numbers for defenses like Progent and CaMeL may not transfer to an attacker who can iterate against the deployed monitor. The systematization through Biba/IFC/reference-monitor primitives gives practitioners a principled vocabulary for comparing and selecting defenses, and the open problems (provenance assignment, in-the-loop task security, implicit flows) map directly onto the hardest unsolved design questions for real agentic pipelines.