Lessons from Penetration Tests on Large-Scale Agent Systems

🕒 Published (v1): 2026-05-26 13:58 UTC · Source: Arxiv · link

Why this paper was selected

Penetration test taxonomy reveals recurring vulnerability classes across large agent systems

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Two 2025 penetration tests against proprietary AI agent products reveal that production-grade agent systems remain vulnerable to prompt injection, terminal tool misuse leading to RCE, and container escape—despite stricter development processes than open-source counterparts. The vulnerabilities are not novel but recur because agent harnesses introduce a fundamentally new threat model: a dynamic, externally-influenceable code-execution entity running inside trusted infrastructure. Existing mitigations (user approval, allowlist/denylist pattern matching, inherited cloud security) are systematically insufficient.

Problem

Prior security research focused on open-source agent frameworks; the security posture of proprietary agent systems under real development rigor was unknown. Separately, the field lacked empirical evidence that commonly deployed mitigations (human-in-the-loop approval, command allowlisting, relying on existing cloud monitoring) actually defend against adversarial exploitation of agentic workflows.

Method

White-box penetration tests (with access to application code and infrastructure diagrams) on two proprietary systems deployed in isolated pre-production environments:

1. GitHub Issue Assistant (Case Study 1): A two-phase localize→edit pipeline using self-hosted LLMs, a file-browsing tool, and a subprocess-based terminal tool restricted by a command allowlist/denylist checked via bashlex/ast. Threat actors modeled as GitHub Contributors (read/issue-create only) and GitHub Owners (write access). Attacks tested: prompt injection via issue text, markdown hiding of malicious instructions, terminal tool exploitation via sed -e and find -ok flags, reverse shell establishment, and lateral data exfiltration.

2. Agent Design Platform (Case Study 2): A chat playground extended with predefined tools, a Python interpreter, and a custom tool builder (user-defined JSON schema + Python implementation). Attacks tested: direct code execution via interpreter, bypassing LLM safety filters through the custom tool builder's debug interface, container escape, and cross-session data leakage from non-ephemeral storage.

Key Contributions

• Empirical demonstration that prompt injection + markdown hiding defeats human-review gates in GitHub Flavored Markdown—malicious instructions are invisible in rendered views but processed by agents reading raw text.
• Proof-of-concept RCE chains through nominally restricted terminal tools: echo a | sed "1e rm -rf ." and echo y | find . -name "exploit.sh" -ok bash {}; bypass allowlists using lesser-known flag semantics.
• Identification that custom tool builder debug interfaces provide a direct code execution primitive that entirely bypasses LLM alignment/safety filtering.
• Structural critique of three widely-deployed mitigations shown to be insufficient: user approval, static allowlist/denylist pattern matching, and inherited cloud security posture.
• Defense-in-depth recommendations operationalized across four layers: sandboxing (seccomp, AppArmor, bwrap, RestrictedPython), fine-grained attribute-based access control (beyond RBAC), I/O sanitization at tool invocation boundaries, and agent-specific observability (thought/action/observation triplet logging).
• Survey of OpenClaw as a reference architecture with allowlisting, runtime isolation, formal verification stubs, and VirusTotal-based code analysis (Table 1).

Results

No quantitative benchmark numbers are reported; findings are qualitative/demonstrative: - Case Study 1: Successful repository integrity compromise (patch injection), reverse shell to attacker-controlled server, and data exfiltration from cloud object storage (cross-repository clones accessible from compromised container). - Case Study 2: RCE via Python interpreter despite LLM identifying code as unsafe, direct code execution via custom tool builder debug interface, access to SSL root certificates/environment variables/internal service metadata, and confirmed cross-session data persistence between container instances. - Both systems: existing cloud monitoring did not flag malicious agent activity because agent tool calls originated from trusted internal service addresses.

Limitations

• Only two case studies; generalizability to the broader proprietary agent ecosystem is limited.
• White-box access (code + infrastructure diagrams) was granted for Case Study 1, accelerating but potentially inflating the severity of findings relative to true black-box attacker conditions.
• Case Study 2 was explicitly a "development playground" with minimal restrictions by design; authors acknowledge this context.
• OpenClaw's formal verification entries in Table 1 are marked with an asterisk indicating the repository was retracted and details are unknown—limiting reproducibility of that reference architecture.
• No comparison against a control group of open-source frameworks using the same methodology is provided.

Relevance to Harnesses / Meta-Harnesses

Harnesses and meta-harnesses are precisely the architecture under attack here: multi-phase orchestration pipelines (localize → judge → edit → publish) with aggregator/judge modules, multi-tool invocation, and external input ingestion are the structural pattern both case studies exploit. The finding that judge modules and aggregation steps inherit the prompt injection surface of all upstream agents is a direct architectural risk for any meta-harness that aggregates LLM outputs before acting. The failure of allowlist-based tool gating—a common pattern in harness tool registries—demonstrates that harness designers cannot treat tool schemas as security boundaries. The recommended controls (sandboxed execution environments per agent, attribute-based tool access control scoped to task context, and thought/action/observation logging) map directly onto the design space of harness security architecture and are actionable requirements for production harness deployment.