Skip to content

SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SEC-bench is the first fully automated benchmarking framework for evaluating LLM agents on real-world software security tasks, built around a multi-agent scaffold (SECVerifier) that automatically reproduces CVE instances and generates gold patches at $0.87/instance. Evaluated on 200 verified CVEs, state-of-the-art code agents (SWE-agent, OpenHands, Aider with Claude 3.7 Sonnet) reach at most 18.0% on PoC generation and 34.0% on vulnerability patching—far below their 60%+ performance on general SE benchmarks.

Problem

Existing security benchmarks rely on CTF challenges (CYBench), small manually-curated CVE sets (CVE-Bench), or recycled datasets with up to 71% label inaccuracy (CVEFixes). None provide fully automated, reproducible PoC artifacts and verified patches at scale, preventing rigorous evaluation of LLM agents on in-the-wild security engineering tasks.

Method

SEC-bench has three modules:

  1. Preprocessor: Collects CVE metadata from the OSV database (38,201 initial instances), scrapes bug reports from GitHub/Bugzilla/Chromium, builds customized per-project Docker environments with sanitizer-enabled harnesses. Filters to 898 candidate instances with available sanitizer reports.

  2. SECVerifier (multi-agent scaffold): A manager agent coordinates three specialized sub-agents sequentially:

  3. Builder: Compiles the vulnerable codebase, resolves dependency issues.
  4. Exploiter: Reproduces or synthesizes a working PoC input that triggers the expected sanitizer report (e.g., AddressSanitizer).

  5. Fixer: Consolidates candidate fix commits into a single patch and validates it eliminates the sanitizer finding without breaking compilation. Sanitizer verdicts serve as a deterministic oracle—no subjective judgment.

  6. Evaluator: Packages verified instances into Docker images with two task formulations: (a) PoC generation given vulnerability description + codebase, (b) vulnerability patching given description + call stack + PoC.

Manual inspection by two senior security engineers (3 rounds) eliminates cases with patch leakage in bug reports or superficial fixes.

Key Contributions

  • First general multi-agent scaffold (SECVerifier) for automatically reproducing real-world CVEs and constructing scalable security benchmarks.
  • Two realistic, execution-verified task formulations (PoC generation, vulnerability patching) grounded in professional security engineering workflows.
  • Comprehensive evaluation of SOTA code agents (SWE-agent, OpenHands, Aider) Ă— models (Claude 3.7 Sonnet, GPT-4o, o3-mini) revealing large performance gaps versus general SE benchmarks.
  • Ablation showing SECVerifier outperforms single-agent CodeAct baseline by 85.7% on verification success (26.0% vs. 14.0%).

Results

  • Verification: SECVerifier successfully verifies 200/898 candidate CVE instances (22.3% overall); sub-agent rates: builder 81.7%, exploiter 39.4% (bottleneck), fixer 69.2%.
  • PoC generation (200-instance full set, Claude 3.7 Sonnet): SWE-agent 12.5%, OpenHands 18.0% (best), Aider 1.2%.
  • Vulnerability patching (200-instance full set, Claude 3.7 Sonnet): SWE-agent 33.8%, OpenHands 34.0% (best), Aider 20.0%.
  • 80-instance subset best: SWE-agent + Claude 3.7 Sonnet achieves 33.8% patch / 12.5% PoC; GPT-4o and o3-mini consistently lower across all scaffolds.
  • Data contamination: No statistically significant difference between pre/post knowledge-cutoff instances (Wilcoxon p=0.27), ruling out memorization as a confound.
  • Failure modes (patching): OpenHands dominated by Improper Format (62.18% of failures due to oversized patches); SWE-agent dominated by Compilation Error and Still Vulnerable; Aider avoids formatting failures via Git integration but suffers No Patch failures.

Limitations

  • Restricted to C/C++ projects; sanitizer-based oracle not applicable to web, kernel, or distributed-systems vulnerabilities.
  • Covers only memory safety vulnerability classes detectable by sanitizers (e.g., AddressSanitizer); broader CWE categories excluded.
  • Low exploiter success rate (39.4%) means many seed instances cannot be verified, reducing final benchmark size.
  • Budget constraints required full 200-instance evaluation to use only the best model; cross-model comparisons are on the 80-instance subset.

Relevance to Agentic AI / LLM Agents

SEC-bench directly stresses-tests the multi-step agentic loop (environment interaction, tool use, iterative debugging) in a domain where failure is objectively measurable, providing a rigorous complement to general SE benchmarks like SWE-bench. The 18–34% ceiling reveals that current agent architectures—even with strong models like Claude 3.7 Sonnet—lack the byte-level reasoning, precise data/control-flow tracing, and debugger integration needed for security tasks, pointing to concrete gaps in tool use and long-horizon planning. The SECVerifier multi-agent design (manager + specialized sub-agents) is itself a generalizable agentic pattern for automated benchmark construction from unstructured real-world data. For researchers building security-aware or autonomous coding agents, SEC-bench establishes a reproducible, scalable evaluation standard that could drive targeted capability improvements.