Rethinking Penetration Testing for AI-Enabled Systems: From Resource Compromise to Behavioral Objective Violation

🕒 Published (v1): 2026-07-15 16:36 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Traditional penetration testing succeeds or fails on whether adversaries can compromise computational resources (credentials, servers, data stores), but AI-enabled systems can be "penetrated" without any infrastructure breach — an adversary who plants instructions in a retrieved document can hijack an LLM agent's tool calls and violate its operational mission. This paper redefines AI-enabled penetration as the feasible induction of AI-governed behavior that violates one or more operational objectives under an explicit threat model, and proposes a six-step objective-driven testing workflow to operationalize that definition.

Problem

Existing penetration-testing standards (NIST SP 800-115, MITRE ATT&CK) and adversarial-ML/AI-red-teaming bodies of work each address important sub-problems but leave a critical gap: none provides a unified success criterion for cases where adversarial influence alters AI-mediated behavior without compromising any conventional resource. A tester following only classical methods may declare a system secure while prompt injection, retrieval poisoning, or sensor manipulation silently violates the system's operational purpose.

Method

The paper introduces a three-layer causal model — (1) computational resources, (2) AI-governed behavior (predictions, generated content, tool calls, plans), (3) operational objectives (accurate triage, safe navigation, compliant decisions) — and argues that adversarial leverage now operates primarily on layer 2 rather than layer 1.

From this model the authors derive two formal definitions: - AI-enabled system: one in which learned models materially influence behavior affecting operational outcomes. - AI-enabled penetration: the feasible induction of AI-governed behavior that violates ≥1 operational objectives under an explicit threat model.

They then propose a six-step objective-driven testing workflow: 1. Identify and scope operational objectives. 2. Map AI-governed behaviors (which model outputs affect which objective). 3. Enumerate adversarial influence surfaces (prompt, retrieval corpus, sensor feeds, tool outputs, memory, training pipeline, human-AI interaction loop). 4. Define behavioral failure criteria (what output or action counts as objective violation). 5. Execute scenario-based tests and collect probabilistic evidence (success rate across repeated trials under defined conditions). 6. Report evidence linking adversarial action → induced behavior → objective violation → operational impact → recommended mitigation.

A running example traces how an indirect prompt injection planted in a retrieved threat-intelligence document causes a SOC-assistant LLM agent to downgrade a high-severity alert, achieving penetration without touching any infrastructure.

Key Contributions

• Conceptual clarification of the three-layer distinction: resource compromise ≠ behavioral compromise ≠ objective violation.
• Formal definition of AI-enabled penetration that extends classical pentest success criteria to behavior-mediated failure modes while preserving conventional infrastructure testing.
• An objective-driven workflow applicable across LLM agents, RAG pipelines, cyber-physical perception systems, biometric authentication, and recommender systems.
• Incorporation of probabilistic evidentiary standards (success rate, reproducibility conditions) suited to stochastic model behavior.
• Mapping of the confused-deputy pattern onto agentic AI: the model/agent is induced to misuse its own legitimate tool authority.

Results

This is a conceptual/framework paper with no empirical benchmarks. The sole illustrative evidence is the SOC-assistant running example demonstrating the indirect-prompt-injection pathway. No quantitative results or comparative experiments are reported.

Limitations

• No empirical validation of the workflow on real AI-enabled systems or red-team exercises.
• The paper is truncated; stated limitations in Section 9 ("open research challenges") are not included in the provided text.
• Defining and scoping "operational objectives" in practice may be ambiguous or contested within an organization.
• Probabilistic success criteria (what success rate constitutes penetration?) are left underspecified; the framework acknowledges this without resolving it.
• Coverage of non-LLM AI systems (vision models, recommenders) is illustrative rather than technically elaborated.

Relevance to Agentic AI / LLM Agents

This paper directly addresses the core threat surface of LLM agent systems: prompt injection, indirect prompt injection, tool misuse, and agentic misalignment are all named attack pathways in the proposed framework. The confused-deputy formalization — where an agent with legitimate tool authority is induced by untrusted retrieved content to act against operator intent — is a precise, theoretically grounded description of the dominant security failure mode in RAG-backed and tool-using agents. For researchers tracking agentic AI, the paper provides a vocabulary and testing workflow that complements attack-focused work (e.g., MITRE ATLAS, OWASP LLM Top 10) by supplying the missing success criterion: not "did the injection execute?" but "did it cause objective violation under a plausible threat model?" The probabilistic evidence standard is particularly relevant given the non-determinism of LLM agent behavior.