Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testing¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR; first live enterprise comparison of AI agents vs. human pentesters (ARTEMIS system)

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper presents the first head-to-head evaluation of AI agents against professional human penetration testers on a live university network of ~8,000 hosts. The authors introduce ARTEMIS, a multi-agent red-teaming scaffold, which placed second overall on a scored leaderboard, outperforming 9 of 10 human professionals. Existing scaffolds (Codex, CyAgent) underperformed most humans, showing that scaffold design—not just model capability—is decisive.

Problem¶

Existing cybersecurity benchmarks (CTF suites, CVE reproduction, Q&A) abstract away the noise, scope, and interactivity of real production environments. Frontier models score ~50% or below on benchmarks like Cybench, yet threat actors are already using AI effectively in real operations, indicating a significant validity gap in current measurement approaches. No comprehensive human-vs-AI comparison had previously been conducted in a live enterprise environment.

Method¶

Ten recruited cybersecurity professionals and six AI agent configurations were each given student-level access to a scoped university network (12 subnets, ~8,000 hosts, heterogeneous Unix/Windows/IoT) and tasked with open-ended penetration testing under IRB safeguards and continuous human oversight. Performance was scored via a unified metric:

\[S_{\text{total}} = \sum_{i=1}^{n} (TC_i + W_i)\]

where $TC_i$ is technical complexity (detection complexity + exploit complexity, with a −0.2 soft penalty for verification-only findings) and $W_i$ is business impact weighting (8/5/3/2/1 for Critical/High/Medium/Low/Informational). ARTEMIS (Automated Red Teaming Engine with Multi-agent Intelligent Supervision) consists of: (1) a supervisor managing workflow via task lists and a note-taking system; (2) a swarm of arbitrary sub-agents with dynamically generated task-specific system prompts; and (3) a triager for vulnerability verification and deduplication. Context longevity is achieved by summarizing progress, clearing context, and resuming across sessions. A2 uses an ensemble supervisor (Claude Sonnet 4, o3, Claude Opus 4, Gemini 2.5 Pro, o3 Pro) with Claude Sonnet 4 sub-agents; A1 uses GPT-5 throughout. Both ran for 16 hours, evaluated on the first 10 for comparability.

Key Contributions¶

First comprehensive live-environment head-to-head evaluation of AI agents vs. human penetration testers at enterprise scale.
ARTEMIS framework: dynamic sub-agent prompt generation, session-based context management, and a triage module for false-positive reduction.
Scoring rubric that rewards technical exploitation depth over "low-hanging fruit," benchmarked against MITRE ATT&CK mappings.
Elicitation trials isolating planning/identification failures from technical execution capability.
Cost-performance analysis: A1 at $18.21/hr vs. ~$60/hr human market rate with competitive output.

Results¶

ARTEMIS A2 placed 2nd overall (total score 95.2), outperforming 9 of 10 human participants; A1 scored 53.2, outperforming ~50% of humans.
Top-ranked human P1 scored 111.4; ARTEMIS A2 at 95.2 was significantly closer to the top human than to A1.
A2 discovered 9 valid vulnerabilities with an 82% valid submission rate.
Human cohort discovered 49 total validated unique vulnerabilities (3–13 per participant).
Claude Code and MAPTA refused the task out of the box, yielding 0 findings; Incalmo stalled at reconnaissance for 0 findings; Codex (GPT-5) scored 38.6, CyAgent+Claude Sonnet 4 scored 23.6.
On Cybench, CyAgent+Claude 4.5 Sonnet achieved the highest reported rate at 55%; ARTEMIS showed no significant uplift over CyAgent+GPT-5 on single-host CTF tasks (attributed to sampling variance, not scaffold failure).
ARTEMIS peaked at 8 concurrent sub-agents, averaging 2.82 per supervisor iteration.
A1 cost $291.47 total ($18.21/hr); A2 cost $944.07 ($59/hr); U.S. median penetration tester salary ~$125,034/year.

Limitations¶

Single-site study (one university network); generalizability to other enterprise environments is untested.
ARTEMIS exhibits higher false-positive rates than human participants.
Agents struggle with GUI-based tasks entirely (e.g., browser-driven interfaces).
ARTEMIS tends to submit findings immediately rather than pivoting to deepen exploitation chains, unlike top human performers.
Elicitation trials reveal ARTEMIS misses vulnerabilities not from technical incapability but from failure to identify the correct vulnerability pattern without hints.
A2's ensemble supervisor is expensive ($59/hr) and its advantage over A1 is partly attributable to broader model coverage, not purely scaffold design.
Human participants had variable active engagement time; P1 performed external reconnaissance before VM provisioning, complicating direct time comparisons.

Relevance to Agentic AI / LLM Agents¶

This paper is a landmark empirical calibration point for agentic LLM capability in a high-stakes, long-horizon, real-world task: it directly quantifies where current multi-agent scaffolds sit relative to expert humans, and shows that scaffold architecture (session management, dynamic prompt generation, triaging) is at least as important as raw model capability—GPT-5 in Codex outperforms only 2 humans, while GPT-5 in ARTEMIS outperforms ~50%. The finding that Claude Code and MAPTA refuse the task while ARTEMIS using the same underlying models does not implicates system-prompt and scaffolding design as a core variable in agentic alignment and capability elicitation research. The cost-efficiency results ($18–59/hr vs. $60+/hr human) concretize the near-term economic disruption potential of capable agentic systems, making this directly relevant to anyone studying agent deployment, capability evaluation methodology, and agentic safety at production scale.