Skip to content

Environment-Grounded Multi-Agent Workflow for Autonomous Penetration Testing

🕒 Published (v1): 2026-03-25 11:50 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper proposes a LangGraph-based three-agent workflow (Planner, Executor, Memory Agent) for autonomous penetration testing of ROS/ROS2 robotic systems, using a shared graph-structured persistent memory ("GraphMemory") anchored to the observed network state. Evaluated on a ROS-based CTF benchmark, the best model configuration (llama-3.3-70b-instruct) achieves 100% success (5/5) across all four CTF tasks, whereas the HackSynth baseline solves none of the ROS-specific exploitation tasks.

Problem

Existing LLM-driven penetration-testing systems (PentestGPT, HackSynth, AutoAttacker) exhibit three compounding gaps: (1) incomplete end-to-end automation—agents cannot generate new tasks dynamically from prior findings; (2) no persistent, decoupled context management—information is either lost between modules or degraded by summarization; (3) insufficient traceability and human oversight to satisfy regulatory requirements such as the EU AI Act. These gaps are especially acute in OT/robotic environments using ROS/ROS2, which have domain-specific attack surfaces (unauthenticated TCP topics, DDS misconfigurations) that general-purpose LLM agents cannot handle reliably.

Method

The system is a closed-loop LangGraph workflow with three cooperating agents:

  • Planner: generates structured JSON task lists (with task_id, task_type, target, dependencies) validated against a predefined schema before passing to the Executor.
  • Executor: translates tasks into concrete CLI commands—Nmap for scanning, custom Bash scripts for ROS exploitation—and runs them against the target.
  • Memory Agent: after each execution, structures outputs into GraphMemory, a persistent graph-based knowledge base keyed by IP address; nodes are created or updated with task-linked entries. Before each planning cycle, the Memory Agent also queries GraphMemory and returns relevant context to the Planner.

Operational modes (Scanning → ROS Exploitation) are user-switched, providing a human-in-the-loop control boundary. Within a mode, agents operate fully autonomously. A predefined deterministic routine captures and validates ROS topic enumeration output, bridging unstructured LLM output to structured script-based validation. The Executor model is fixed at llama-3.1-8b-instruct; the Planner is varied across llama-3.3-70b-instruct, deepseek-v3.2, gemma-3-27b-it, and hermes-2-pro-llama-3-8b.

Key Contributions

  • A graph-based persistent shared memory (GraphMemory) that incrementally captures network topology, executed actions, and outcomes—enabling full attack-path reconstruction and satisfying EU AI Act traceability requirements.
  • An environment-grounded multi-agent architecture with explicit mode boundaries and configuration matrices that constrain valid IP ranges and tool usage, improving controllability without sacrificing autonomous reasoning.
  • Empirical evaluation against the MAST failure taxonomy on a Docker-based ROS/ROS2 CTF testbed, with systematic comparison to HackSynth across four LLM backends.
  • Failure mode analysis mapped to MAST categories: Reasoning–Action Mismatch (2.6), Step Repetition (1.3), and Task Derailment (2.3).

Results

  • With llama-3.3-70b-instruct as Planner, the proposed system achieves (5/5) on CTF-0, CTF-1, CTF-2, and CTF-3—100% across all tasks and all runs.
  • deepseek-v3.2 solves CTF-0 (5/5) and CTF-2 (1/5) but fails CTF-1 (1/5) and CTF-3 (0/5).
  • gemma-3-27b-it and hermes-2-pro-llama-3-8b solve CTF-0 (5/5) but fail CTF-1–3 (0/5 each).
  • HackSynth baseline: solves CTF-0 reliably for strong models, partially solves CTF-1, and achieves 0/5 on CTF-2 and CTF-3 across all evaluated models.
  • The proposed system's 100% completion rate "significantly exceeds literature benchmarks" (direct quote; no aggregate baseline percentage given in the text).

Limitations

  • Evaluation uses only \(n=5\) runs per configuration on a single, purpose-built CTF scenario—generalization to real-world or diverse OT environments is unvalidated.
  • Code is not publicly available (available from authors upon request), limiting reproducibility.
  • Agents occasionally generate overly complex Nmap commands, inflating scan duration; the paper notes this could be mitigated by injecting tool documentation into the planning phase but does not implement it.
  • Mode switching remains manual (human-in-the-loop), so the system is not fully autonomous end-to-end.
  • Smaller models (hermes-2-pro-llama-3-8b) fail at CTF-1, gating all downstream exploitation—model capability remains a hard dependency.
  • Evaluation scope is restricted to ROS 1/ROS 2 reconnaissance and topic-level exploitation; deeper exploit chains or post-exploitation are not tested.

Relevance to Harnesses / Meta-Harnesses

This paper is a concrete instantiation of a domain-specialized agentic harness: a LangGraph orchestration layer that wires together a Planner, Executor, and Memory Agent into a reproducible closed-loop workflow, directly analogous to how meta-harnesses wrap and sequence specialized sub-agents. The GraphMemory construct—a persistent, structured, incrementally-updated shared state decoupled from individual agent contexts—addresses the same coordination and context-persistence problems that meta-harnesses solve in general-purpose pipelines. The explicit mode-boundary mechanism (user-switched operational modes with constrained tool sets) is a practical pattern for scoping agent authority within a harness, relevant to anyone designing harnesses where agents must operate with guardrails. The MAST-grounded failure analysis also provides a vocabulary for diagnosing multi-agent harness failures (task derailment, step repetition, reasoning–action mismatch) transferable to non-security harness contexts.