Skip to content

Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents

🕒 Published (v1): 2026-07-16 17:54 UTC · Source: Arxiv · link

Why this paper was selected

Singer (Harvard); reframes security-agent eval as cost-aware rather than peak-capability — changes how benchmarks are designed

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper argues that evaluating security agents by peak success rate alone is insufficient, and proposes measuring cost-success tradeoffs instead. It benchmarks LLM agents on offensive CTF (Cybench) and defensive SOC investigation (Splunk BOTS v1) tasks, finding fundamentally different scaling behaviors between the two task types.

Problem

Existing security-agent benchmarks measure peak capability under generous inference budgets, ignoring the operational reality that every reasoning step, tool call, and enrichment request consumes real budget. This makes it impossible to assess which models are economically viable in production SOC environments, and conflates two distinct cost components: model-token spend and priced external-tool spend.

Method

The authors implement a unified evaluation harness using the Inspect framework with a ReAct-style agent and auto-compaction (triggered at 90% of context window). They run agents on: - Cybench hard variant (39 CTF challenges; tools: bash, python, submit; all zero marginal cost) - BOTS v1 (31 scored Splunk incident-investigation questions worth 10,300 pts; tools: Splunk SPL, VirusTotal, WHOIS, Brave Search, DNS, bash, python; external enrichment priced per Table 1)

Cost caps are applied per-sample via Inspect's budget option; exceeded samples are scored as failed. Retrospective budget-headroom analyses replay completed traces under a $0.80/sample cap to isolate scaling behavior without confounding model choice. Contamination controls include no-tools probes (with and without prerequisite Q&A context) to detect BOTS v1 leakage into model training data.

Key Contributions

  • A cost-success comparison framework treating per-sample dollar budget as the common axis across offensive and defensive security tasks
  • Paired retrospective budget-headroom analysis (same model, same trace, lower cap applied post-hoc) isolating scaling behavior
  • Decomposed cost accounting: model-token spend + priced external-tool spend (Brave Search $0.005/req, WhoisXMLAPI preview $0.0258/req, purchase $1.29/req)
  • Cost-efficiency metric: $/solved-challenge (Cybench) and $/1,000 BOTS points
  • Contamination controls for BOTS v1 showing strongly model-dependent no-tools answer recovery (13–77% range), making decontamination checks mandatory for public SOC benchmarks
  • Interactive results website at evals.frontier.security

Results

Cybench (offensive): - GPT-5.5: 94.1% success, $1.16/solve (most capable) - DeepSeek v4 Flash: 86.4% at $2.10 cap, 76.1% at $0.80 cap — \(1.45/solve, most cost-efficient at full budget (\)1.03/solve at $0.80) - Claude Opus 4.8: 76.2%, $3.18/solve - Claude Fable 5: 0% (refused all 117 sample-epochs); GPT-5.6 Sol: 9.4% (refused 106/117) - Budget scaling is real: Opus 4.8 gains +18.8 pp [10.3, 29.1] moving from $0.80→full cap; DeepSeek v4 Flash gains +10.3 pp [4.3, 17.1]

BOTS v1 (defensive): - Claude Opus 4.8: 9,666.7/10,300 pts (93.9%), \(2.98/1k pts, 603 total tool calls — top performer and most cost-efficient - GPT-5.6 Terra: 92.1% (\)3.55/1k pts, 1,567 tool calls); Sol: 91.4% ($4.70/1k pts) - Claude Fable 5: 88.4% with only 309 tool calls and 5/93 refusal events - DeepSeek v4 Flash: doubling cap from \(2.10→\)4.20 yields only +0.9 pp (73.0%→73.9%), while tool calls climb 4,450→4,938 — extra budget bought no meaningful improvement - Budget scaling is weak: Opus 4.8 gains only +2.6 pp above $0.80; Flash gains +0.0 pp - Contamination: no-tools probes yield 50–77% for some models (GPT-5.6 Sol, Opus 4.8), vs. 13–18% for others, indicating model-specific leakage

Limitations

  • BOTS v1 is public and old; no-tools probe results show probable benchmark contamination, so absolute BOTS scores cannot be cleanly interpreted as live investigation skill
  • Retrospective budget-headroom analysis is descriptive (replaying completed traces), not a new prospective run, so it cannot capture strategies an agent would choose differently under a tighter budget
  • Reported dollar costs exclude infrastructure (Kubernetes, Splunk, storage) and analyst review time, so real TCO is higher
  • Refusal detection uses heuristic text matching, not provider policy labels, which may misclassify edge cases
  • DeepSeek v4 Pro's near-zero Cybench budget-headroom delta is confounded by its low prospective cap, not necessarily evidence of poor scaling
  • Only two benchmark domains tested; findings may not generalize beyond CTF and SOC investigation task types

Relevance to Agentic AI / LLM Agents

This work directly addresses a blind spot in agent evaluation methodology: reporting peak accuracy without cost constraints misrepresents operational utility. The finding that offensive and defensive tasks exhibit fundamentally different scaling regimes — offensive success rises monotonically with compute, while defensive success saturates and depends on tool discipline — is a concrete empirical result that challenges the assumption that more test-time compute universally improves agentic performance. For researchers building or evaluating LLM agents, the decomposition of inference spend versus priced-tool spend, and the use of retrospective budget caps as a controlled scaling analysis, are reusable methodological contributions. The contamination controls also highlight a broadly relevant problem: public agentic benchmarks are vulnerable to training-data leakage in ways that vary by model and corrupt leaderboard comparisons.