Skip to content

DefenderBench: A Toolkit for Evaluating Language Agents in Cybersecurity Environments

🕒 Published (v1): 2025-05-31 23:00 UTC · Source: Arxiv · Venue: NEURIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DefenderBench is an open-source Python toolkit that wraps five cybersecurity task environments (network intrusion simulation, malicious content detection, code vulnerability detection/fixing, and CTI question answering) into a unified agentic evaluation harness. It defines a composite DefenderBench score (unweighted average across tasks) and benchmarks 20+ LLMs. Claude-3.7-sonnet achieves the best aggregate score of 81.65.

Problem

Existing cybersecurity LLM benchmarks (Cybench, CyberMetric, CyberSecEval) each cover only one or two task types, making cross-model comparisons narrow and preventing holistic assessment of LLM agents across the full offense/defense/knowledge spectrum. No prior toolkit provided a reproducible, modular framework for adding custom LLMs and tasks under a single scoring rubric.

Method

DefenderBench is implemented as an installable Python library with three coupled modules:

  1. Data Preprocessing — automatically downloads datasets (CyberBattleSim, phishing corpora, CodeXGLUE/Devign, CVEFix, CTI-MCQA), applies a fixed random seed shuffle, caches locally, and constructs few-shot ICL pools.
  2. Task Environments — each environment exposes a text-action/observation loop. Detection and QA tasks run episodes of up to 5 steps with format-feedback retries; network intrusion (CyberBattleSim converted to a JSON text game) runs up to 100 steps. Task-specific instructions are injected as system prompts (see Table 1).
  3. Agent Interface — a minimal baseline agent maintains a trajectory history in context and issues one action per step. Users plug in any open- or closed-weight LLM via the interface; a plugin system allows adding new tasks. Weights & Biases is integrated for logging.

The global metric is the unweighted average of per-task metrics (win rate, Macro-F1, or CodeBLEU) called the DefenderBench score.

Key Contributions

  • Open-source toolkit that unifies five cybersecurity task types (offense + defense + knowledge) under one evaluation pipeline.
  • Modular plugin architecture enabling seamless addition of custom LLMs, agents, and tasks.
  • Standardized baseline agent with minimal scaffolding evaluated across 20+ LLMs (open-weight and proprietary).
  • Composite DefenderBench score aggregating heterogeneous metrics for fair cross-model comparison.
  • Auxiliary analyses of chain-of-thought prompting and augmented information on task subsets.

Results

  • Claude-3.7-sonnet: 81.65 DefenderBench score (best overall); 100% win rate on both CBS-Chain and CBS-CTF.
  • Claude-3.7-sonnet-think: 78.40 (best reasoning model); reasoning models do not consistently outperform non-reasoning counterparts.
  • Llama 3.3 70B: 71.81 (best open-weight), outperforming GPT-3.5 (52.99).
  • Network intrusion (CBS-Chain): Llama 3.3 70B, Claude-3.5-sonnet, Claude-3.7-sonnet all achieve 100% win rate; Llama 3.3 70B is most efficient at avg. 26.5 steps vs. Claude-3.7-sonnet at 50.2.
  • Vulnerability detection: All models hover near random baseline; best Macro-F1 is GPT-4-turbo at 58.20 (Vuln.-CG) and o3 at 59.60 (Vuln.-DV).
  • CVEFix: No LLM exceeds the copy-paste baseline in CodeBLEU; GPT-4o-mini scores highest at 79.71.
  • CoT prompting: Mixed effect — improves interactive tasks slightly for some models (e.g., GPT-4o CBS win rate 56.3→73.3%) but slightly degrades static classification tasks.
  • Larger models consistently outperform smaller ones (Llama 3.1 70B beats 8B by 13.97 points; GPT-4.1 > GPT-4.1-mini > GPT-4.1-nano: 63.90 > 58.90 > 47.50).

Limitations

  • CodeBLEU mismatch: For patch-style vulnerability fixes involving minimal edits, CodeBLEU compares entire methods and does not meaningfully reflect patch quality; no LLM agent beats the copy-paste baseline.
  • Subsampled test sets: Most tasks use 500 randomly sampled test items, which may not fully represent the underlying distribution or task difficulty.
  • Minimal agent scaffolding: The baseline agent has no tool use, memory, or planning; results may underestimate what more sophisticated agentic frameworks could achieve.
  • Static environments: Task configurations are fixed; the benchmark does not evaluate agents in dynamically evolving or adversarial environments.
  • No execution-based vulnerability validation: Code vulnerability detection relies on supervised labels rather than dynamic analysis or fuzzing.
  • Single-turn few-shot only: ICL pool is fixed at 10 samples per class; retrieval-augmented or adaptive prompting is not explored in the main benchmark.

Relevance to Harnesses / Meta-Harnesses

DefenderBench is a domain-specific evaluation harness that demonstrates the canonical meta-harness design pattern: a unified driver that wraps heterogeneous task environments behind a common agent interface, automates data procurement and preprocessing, injects standardized prompts, collects multi-metric results, and computes an aggregate score — all orchestrated by a single CLI entry point. Its modular plugin system for tasks and LLMs is directly analogous to how meta-harnesses abstract over multiple downstream pipelines. The work is relevant as a concrete instantiation of how harness architecture choices (episode termination, history injection, format-retry loops, fixed seeds for reproducibility) affect evaluation validity, providing a replicable template for researchers building harnesses in other domains.