Systematic Capability Benchmarking of Frontier Large Language Models for Offensive Cyber Tasks¶
🕒 Published (v1): 2026-04-18 22:13 UTC · Source: Arxiv · link
Why this paper was selected
Most comprehensive cross-model eval of LLM agents on offensive cyber tasks
Ask a follow-up
Open an assistant pre-loaded with this paper's context.
💬 Ask ChatGPT✦ Ask Claude
TL;DR¶
This paper extends the D-CIPHER multi-agent framework into a reproducible evaluation harness that benchmarks 10 frontier LLMs across 200 CTF challenges under controlled factorial conditions. Key findings: environment tooling (Kali vs Ubuntu) and model selection dominate performance, while prompt engineering interventions degrade results in well-equipped setups. Claude 4.5 Opus achieves 59% solve rate; Gemini 3 Flash offers the best cost-efficiency at $0.05/solve.
Problem¶
Prior LLM-on-CTF studies are incomparable: they differ in model selection, execution environment, prompting strategy, and evaluation protocol, making it impossible to attribute performance differences to any single engineering factor. No prior work systematically ablates these dimensions on the same benchmark under controlled conditions.
Method¶
The authors extend D-CIPHER—a hierarchical Planner–Executor multi-agent system—with three additions: (1) multi-provider backend support (Vertex AI, OpenRouter, Ollama) enabling cross-provider evaluation; (2) a custom Kali Linux Docker image with 100+ pre-installed penetration testing tools; (3) runtime tool-discovery agents (ListCommandsTool, LookupCommandTool) backed by a structured commands_documentation.csv.
A controlled factorial study varying three binary factors (environment: Ubuntu/Kali × prompts: generic/category-tips × auto-prompting: off/on) is run with Gemini 3 Pro across all 1,600 configuration-challenge pairs. The best configuration is then used for cross-model comparison (10 models, 7 providers) and multi-agent architecture analysis (symmetric vs. asymmetric Planner/Executor tier assignment). Hyperparameters are fixed: \(T=1.0\), 30 planner rounds, 100 executor rounds, 10-minute wall-clock timeout, $5.00/challenge cost ceiling.
Key Contributions¶
- Systematic cross-model evaluation of 10 frontier LLMs on all 200 NYU CTF Bench challenges—the most comprehensive such comparison to date.
- Controlled ablation showing Kali Linux yields +9.5 pp over Ubuntu under matched prompts; auto-prompting reduces solve rate by up to −28.5 pp in well-equipped environments.
- Cost-performance analysis: Claude 4.5 Opus tops at 59.0% (118/200); Gemini 3 Flash achieves 27.0% at \(0.05/solve (best efficiency); Gemini 3 Pro at 52.0%/\)0.43/solve.
- Reproducible infrastructure: Kali Docker image, runtime tool-discovery agents, parallel experiment runner, automated result-parsing pipeline (code to be released).
- Finding that homogeneous planner/executor configurations (same model both roles) consistently outperform asymmetric mixed-tier pairings.
Results¶
- Environment: Kali + Generic (no AutoPrompt): 52.0% vs. Ubuntu + Generic: 42.5% (+9.5 pp), with Gemini 3 Pro.
- Auto-prompting penalty: Kali + Generic + AutoPrompt drops to 39.5% (−12.5 pp); Ubuntu + Tips + AutoPrompt collapses to 16.0% (−28.5 pp).
- Model ranking (Kali + Generic, no AutoPrompt, 200 challenges):
| Model | Solve Rate | $/Solve |
|---|---|---|
| Claude 4.5 Opus | 59.0% | $2.12 |
| Gemini 3 Pro | 52.0% | $0.43 |
| Gemini 3 Flash | 27.0% | $0.05 |
| GLM-5 | 19.5% | $0.57 |
| GPT-5.2-Codex | 18.0% | $0.26 |
| GPT-5.2 | 13.5% | $0.28 |
| DeepSeek-V3 | 6.5% | $0.01 |
| DeepSeek-R1 | 4.0% | $0.02 |
| Qwen 3.5 397B-A17B | 3.5% | $0.20 |
| Llama 3.3 70B | 2.5% | $0.01 |
- Prior SOTA (D-CIPHER, CRAKEN with Claude 3.5 Sonnet): 22%; Ubuntu baseline with Gemini 3 Pro (42.5%) already exceeds this, suggesting model capability gains dominate framework gains.
- Architecture: Pro+Pro: 52.0%; Pro+Flash: 28.5%; Flash+Flash: 27.0%; Flash+Pro: 23.5%—symmetric same-model is dominant; asymmetric provides no meaningful benefit.
- Hardest category: binary exploitation (pwn); easiest: miscellaneous.
Limitations¶
- All experiments are single-trial at \(T=1.0\); results are indicative, not statistically bounded across runs.
- Results are specific to D-CIPHER; other agent architectures may exhibit different factor sensitivities.
- NYU CTF Bench uses historical CSAW competitions (2016–2024) and is susceptible to data contamination; absolute solve rates may be inflated, though relative comparisons under identical conditions remain valid.
- Architecture findings (coherence effect of same-model pairing) are based on only two models from the same provider (Google); cross-provider generalization is untested.
- Benchmark reflects academic CTF tasks, not live adversarial conditions; external validity to real-world offensive operations is uncertain.
Relevance to Harnesses / Meta-Harnesses¶
This work is a direct example of a meta-harness: a framework-of-frameworks that wraps D-CIPHER with multi-provider backends, containerized execution environments, tool-discovery agents, a parallel experiment runner, and automated result parsing—all to enable systematic, reproducible factorial evaluation of sub-harness configurations. The finding that environment tooling (what the harness provides to agents) dominates prompt engineering (what the harness injects as text) is a key design principle for anyone building evaluation or task-solving harnesses. The tool-discovery layer (ListCommandsTool/LookupCommandTool) exemplifies a meta-harness pattern: dynamically surfacing available capabilities to sub-agents at runtime rather than hard-coding tool catalogs. The demonstration that homogeneous same-model planner/executor configurations outperform mixed-tier ones has direct implications for harness designers choosing how to allocate models across roles in orchestrated pipelines.