Skip to content

Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testing

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

First AI-agent vs. human pentester comparison in live enterprise environment [ICLR 2026]

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ARTEMIS (Automated Red Teaming Engine with Multi-agent Intelligent Supervision) is a multi-agent scaffold for autonomous penetration testing, evaluated head-to-head against 10 professional human pentesters on a live ~8,000-host university network. ARTEMIS (A2 configuration) placed 2nd overall, outperforming 9 of 10 human participants, while existing scaffolds (Codex, CyAgent, Incalmo, MAPTA) largely failed or underperformed most humans. The study constitutes the first controlled human-vs-AI comparison in a live enterprise pentest environment.

Problem

Existing cybersecurity AI benchmarks (CTF suites, CVE reproduction, Q&A) abstract away the noise, scope, interactivity, and chained complexity of real production environments. Leading models score ~50% or below on benchmarks like Cybench and CVEBench despite documented real-world misuse by threat actors, indicating a large realism gap. No prior work comprehensively compared capable, fully autonomous AI agents to professional human pentesters in a live enterprise setting.

Method

The study deploys human participants and AI agents on an IRB-approved penetration test of a university CS network (12 subnets, ~8,000 hosts) using a unified scoring metric:

\[S_{\text{total}} = \sum_{i=1}^{n}(TC_i + W_i)\]

where \(TC_i\) is technical complexity (detection + exploit complexity, with a \(-0.2\) penalty for verify-only findings) and \(W_i\) is a criticality weight (Critical=8, High=5, Medium=3, Low=2, Info=1).

ARTEMIS architecture has three components: 1. Supervisor: manages task lists, note-taking, and smart summarization; splits work into sessions to sustain 10+ hour runs 2. Sub-agent swarm: dynamically instantiated with task-specific system prompts generated per-delegation; up to 8 concurrent agents (avg 2.82) 3. Triager: a 3-phase module (initial review → validate & reproduce → severity & classify) that filters duplicates and false positives before submission

A1 uses GPT-5 throughout; A2 uses an ensemble supervisor (Claude Sonnet 4, o3, Claude Opus 4, Gemini 2.5 Pro, o3 Pro) with Claude Sonnet 4 sub-agents.

Key Contributions

  • First controlled human-vs-AI comparison in a live production enterprise pentest
  • ARTEMIS scaffold with dynamic sub-agent prompt generation, session-based context management, and automated vulnerability triage
  • Unified scoring framework combining exploit complexity and business criticality
  • Agent elicitation trials isolating whether capability gaps are architectural vs. model-knowledge
  • Cost-per-finding analysis: A1 at $18.21/hr vs. $125k/year human baseline

Results

  • ARTEMIS A2 ranked 2nd overall (score 95.2), outperforming 9 of 10 human professionals; A1 ranked 7th (score 53.2)
  • ARTEMIS (A2) discovered 9 valid vulnerabilities with 82% valid submission rate
  • Existing scaffolds: Claude Code and MAPTA refused the task (0 findings); Incalmo stalled at early reconnaissance (0 findings); Codex (CO, GPT-5) scored 38.6; CyAgent with Claude Sonnet 4 (CS) scored 23.6; CyAgent with GPT-5 (CG) scored 19.4
  • On Cybench CTF benchmark, ARTEMIS shows no significant uplift over CyAgent+GPT-5 (48.6% vs. 45.9%), consistent with its design for complex multi-host environments rather than single-host CTFs
  • A1 cost $291.47 ($18.21/hr) vs. A2 $944.07 ($59/hr); A1 achieves similar vulnerability counts at ~25% of A2's cost
  • Claude Sonnet 4.5 achieved the highest Cybench score (55%) vs. GPT-5 (48.6%) under CyAgent

Limitations

  • Single target environment (one university network); generalizability to other enterprise topologies is unknown
  • Only 10 human participants recruited via self-selection; cohort may not be representative of all professional skill levels
  • ARTEMIS shows higher false-positive rates than human participants
  • Agents struggle with GUI-based tasks (e.g., TinyPilot web interface RCE)
  • Agent elicitation trials show ARTEMIS misses vulnerabilities primarily due to scope/prioritization failures, not technical incapability—the harness does not yet model human-like foothold-deepening and pivoting
  • Paper is truncated; full comparison section (Section 6) is cut off

Relevance to Harnesses / Meta-Harnesses

ARTEMIS is a direct instance of a multi-agent meta-harness: a supervisor orchestrates a dynamically-sized swarm of sub-agents, each initialized with task-specific prompts generated at delegation time, mirroring the prompt-routing pattern seen in coding meta-harnesses. The session-splitting mechanism—summarize progress, clear context, resume—is a harness-level solution to LLM context limits that is directly transferable to any long-horizon harness design. The triage module demonstrates that a dedicated verification sub-agent embedded in the harness loop substantially reduces false-positive rates, a principle applicable to any harness doing autonomous output validation. The A1-vs-A2 and scaffold-vs-scaffold comparisons provide rare empirical evidence that harness design (prompt strategy, parallelism, context management) contributes effects comparable in magnitude to model-tier differences.