Skip to content

Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testing

🕒 Published (v1): 2025-12-10 18:12 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

The first head-to-head evaluation of AI agents versus human cybersecurity professionals on a live university network (~8,000 hosts). ARTEMIS, a new multi-agent scaffold built for this study, placed second overall, outperforming 9 of 10 human participants while costing as little as $18/hr. Existing scaffolds (Codex, CyAgent, Claude Code out-of-box) underperformed most humans.

Problem

Existing AI cybersecurity benchmarks (CTFs, CVE reproduction, Q&A) lack the scope, noise, interactivity, and CIA constraints of real production environments, causing a mismatch between benchmark scores (~50% on Cybench) and reported real-world misuse. No prior work had compared capable AI agents directly against human penetration testers in a live enterprise setting.

Method

A controlled study on a large research university's CS network (12 subnets, ~8,000 hosts) where 10 recruited professionals and 6+ AI agent configurations performed simultaneous penetration tests under identical instructions, each given up to 10 hours of active engagement. Vulnerabilities were scored via a unified metric combining technical complexity (detection complexity + exploit complexity, with a −0.2 penalty for verify-only findings) and business impact weighting (criticality-scaled multipliers mirroring bug-bounty exponential payouts). ARTEMIS is the authors' novel scaffold: a supervisor LLM orchestrates a dynamic swarm of sub-agents with task-specific system prompts generated on the fly, a triage module filters false positives and duplicates, and a session-splitting mechanism (summarize → clear context → resume) enables 16-hour continuous runs. A2 uses an ensemble supervisor (Claude Sonnet 4, o3, Claude Opus 4, Gemini 2.5 Pro, o3 Pro) with Claude Sonnet 4 sub-agents; A1 uses GPT-5 throughout.

Key Contributions

  • First comprehensive human-vs-AI penetration test in a live production enterprise environment.
  • ARTEMIS scaffold: dynamic sub-agent prompt generation, parallel swarm execution (peak 8 concurrent sub-agents, avg 2.82/iteration), automatic vulnerability triaging, and long-context session splitting.
  • Unified scoring framework combining technical complexity (exploit vs. verify) and severity weighting.
  • Elicitation trials with five hint levels to isolate capability gaps from exploration failures.
  • Cost analysis: A1 at \(18.21/hr (\)37,876/year) vs. average U.S. pentester at $125,034/year; A2 at $59/hr.
  • Open-source release of ARTEMIS and study artifacts.

Results

  • ARTEMIS A2 placed 2nd overall (score 95.2), A1 placed 6th (score 53.2), outperforming 9/10 and 5/10 humans respectively; top human P1 scored 111.4.
  • A2: 11 total findings, 82% valid submission rate; A1: 11 findings, 55% valid.
  • Codex (GPT-5), CyAgent (GPT-5/Claude Sonnet 4), Claude Code, MAPTA, and Incalmo all scored below median human; Claude Code and MAPTA found 0 findings (refused or stalled).
  • On Cybench CTF: ARTEMIS shows no significant scaffold uplift over CyAgent+GPT-5; best model (Claude Sonnet 4 in CyAgent) reaches 55% pass@1 — scaffold effects are environment-dependent.
  • 80% of humans vs. 0% of agents (without hints) found the TinyPilot RCE (GUI-dependent); ARTEMIS uniquely found an old IDRAC vulnerability missed by all humans (obsolete HTTPS cipher, CLI-accessible via curl -k).
  • ARTEMIS submits more false positives than humans (confirmed quantitatively in severity distribution figure).

Limitations

  • Compressed time frame (≤10 active hours, 4 days access) versus typical 1–2 week engagements.
  • Aware IT staff manually approved borderline agent actions; real defensive conditions absent.
  • Small sample (n=10 humans, n=7 agent configs); insufficient statistical power for hypothesis testing.
  • No replicable environment: live target cannot be re-run, precluding ablation studies.
  • ARTEMIS cannot interact with browser GUIs, causing systematic blind spots on GUI-dependent vulnerabilities.
  • ARTEMIS's refusal bypass is scaffold/prompt-driven, raising dual-use concerns not fully resolved by open-sourcing rationale.

Relevance to Harnesses / Meta-Harnesses

ARTEMIS is a direct example of a meta-harness: a supervisor agent that dynamically instantiates specialized sub-agents with generated system prompts, manages long-horizon state via session checkpointing, and routes outputs through a triage module — exactly the architecture pattern central to this research track. The finding that scaffold choice (not model alone) determines whether refusals are triggered or bypassed is highly actionable: harness designers must account for model safety policies as first-class constraints. The demonstrated gap between CTF benchmark performance and live-environment performance reinforces that meta-harness design must be evaluated in realistic settings, not on synthetic tasks. The cost-performance tradeoff analysis (A1 vs. A2) provides a concrete template for multi-model ensemble orchestration decisions in production harnesses.