Skip to content

The Containment Gap: How Deployed Agentic AI Frameworks Fail Public-Facing Safety Requirements

🕒 Published (v1): 2026-06-11 01:46 UTC · Source: Arxiv · Venue: ICML 2026 · link

Why this paper was selected

ICML 2026; systematic audit of agentic framework safety gaps in public-facing deployments

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper audits three dominant agentic AI frameworks (LangChain, AutoGPT, OpenAI Agents SDK) against six formal containment principles and finds zero native compliance in any of them. A simulated government benefits agent demonstrates that a single memory-poisoning write achieves 100% targeted corruption across five model backends, increasing wrongful denial rates to 88.9% while preserving aggregate accuracy. Two deterministic, sub-millisecond fixes (a memory integrity validator and a policy gate) eliminate both attack vectors completely.

Problem

Agentic LLM frameworks composing perception → reasoning → execution → memory pipelines lack architectural-level structural safety guarantees. Prior work catalogs attack types but does not explain why vulnerabilities persist across frameworks and backends; the root cause is the absence of enforced containment at inter-layer boundaries, not model-level deficiencies. No prior work operationalizes formal containment principles into a reusable compliance audit for deployed frameworks.

Method

The authors derive six containment principles (P1–P6: Reasoning-Execution Separation, Capability Scoping, Memory Integrity, Layer-Transition Validation, Authenticated Communication, Runtime Monitoring) from a compositional pipeline model \(\Phi(o_t, m_t) = E(B(P(o_t), m_t))\). They score three frameworks via documentation review and source code inspection (Cohen's \(\kappa = 0.81\)) on a rubric: ✓ native default / ✓* requires configuration / ✗ absent.

Empirically, they build a LangChain welfare-benefits agent (Qwen-2.5 3B, 250 synthetic claims, deterministic eligibility rule) and inject a single adversarial memory write at claim 11 of 40. They measure corruption rate (fraction of targeted post-poison decisions that flip) and accuracy across three seeds and five model backends. Two lightweight interventions are implemented and evaluated:

  1. Memory Integrity Validator (P3): Deterministic regex interposed on ConversationBufferMemory.save_context; checks provenance source (EXTERNAL vs. AGENT), JSON schema conformance, and demographic-targeting patterns; rejected writes are silently dropped.
  2. Tool-Call Policy Gate (P1/P2): Deny-all allowlist over tool names; path canonicalization (resolving ../) over file arguments before execution.

A complex five-factor conjunctive policy experiment (Claude Sonnet 4.6, GPT-4o-mini) tests whether policy complexity enables covert, aggregate-accuracy-preserving attacks.

Key Contributions

  • First reusable compliance matrix operationalizing formal containment principles for agentic framework audit (18 framework–principle dyads).
  • Empirical proof that zero native compliance is not merely theoretical: a single memory-poisoning write achieves corruption rate 1.000 across all tested seeds and all five model backends (Qwen-2.5 3B, Claude Haiku 4.5, GPT-4o, Claude Sonnet 4.6, GPT-4o-mini).
  • Demonstration that the attack is covert under complex policy: overall accuracy remains near baseline while targeted wrongful denials increase \(3.5\times\), evading standard aggregate monitoring.
  • Two deterministic, LLM-free containment mechanisms that reduce both attack vectors from 100% success to 0% with \(< 0.2\) ms overhead, requiring no upstream framework changes.
  • Formal sufficiency theorem: P1 + P3 jointly guarantee that no single-step memory-poisoning attack can induce a persistent policy violation.

Results

  • Baseline accuracy (clean, no attack): 0.908 mean across seeds (Qwen-2.5 3B).
  • Memory poisoning (no guard): Corruption rate = 1.000 across all 3 seeds; mean accuracy collapses from 0.908 → 0.558; Region B wrongful denial rate = 0.889.
  • Memory poisoning generalizes across scale: Claude Haiku 4.5 and GPT-4o both achieve corruption rate 1.000; poisoned accuracy 0.875 (vs. 0.558 for 3B—better on non-targeted claims, not on the attack itself).
  • Memory validator recovery: Corruption rate drops 1.000 → 0.000 across all backends; accuracy recovers to 0.967–1.000; validator overhead = 0.006–0.016 ms per call.
  • Tool-access bypass (no gate): 100% bypass rate across all 100 adversarial prompts (50 path traversal, 25 unauthorized API, 25 restricted write).
  • Policy gate recovery: Bypass rate drops to 0.000 across all three attack types; gate overhead = 0.095–0.129 ms.
  • Complex five-factor policy (Claude Sonnet 4.6): Overall accuracy 0.925 (clean) → 0.925 (poisoned), concealing targeted harm; Region B denials 0.000 → 0.750; validator restores corruption to 0.000.
  • Compliance matrix: No framework achieves ✓ (native default) on any of the six principles; AutoGPT is worst (0✓ / 1✓* / 5✗).

Limitations

  • Compliance rubric captures mechanism presence, not implementation depth or runtime effectiveness.
  • Audit is point-in-time; frameworks may evolve.
  • Memory validator uses deterministic regex; fragile to natural-language adversarial rephrasing of targeting patterns.
  • Policy gate's deny-all allowlist may not generalize to dynamic multi-agent systems with open-ended tool sets.
  • Empirical replication covers only LangChain as the runtime; AutoGPT and OpenAI Agents SDK receive audit-only treatment, no empirical validation.
  • Compound attacks (individually harmless operations with adverse combined effect) require trajectory analysis (P6), which remains an open problem.
  • Evaluation uses synthetic claims and a local adversary model, not real-world adaptive adversaries.

Relevance to Harnesses / Meta-Harnesses

This paper is directly relevant to anyone building or evaluating agentic harnesses: it provides the first formal compliance checklist (P1–P6) for auditing whether a harness enforces containment at its inter-layer boundaries, which is precisely the structural property that distinguishes a safe harness from one that merely chains LLM calls. The finding that safety mechanisms are universally opt-in rather than default—even in production-grade frameworks—has direct implications for meta-harness design: a meta-harness that orchestrates sub-agents across LangChain or similar frameworks cannot inherit safety from them and must inject its own containment gates. The two proposed primitives (memory integrity validator, policy gate) are concretely implementable as harness-layer wrappers with negligible overhead, offering a template for secure-by-default harness architecture.