Skip to content

CyberEvolver: Structured Self-Evolution for Cybersecurity Agents On the Fly

🕒 Published (v1): 2026-05-25 16:26 UTC · Source: Arxiv · link

Why this paper was selected

CyberEvolver: self-evolving cybersecurity agent adapts scaffold on-the-fly to targets

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CyberEvolver is a self-evolving cybersecurity agent framework that iteratively rewrites its own scaffold (prompts, tools, interaction rules, skill library) using structured feedback from failed execution attempts. Unlike fixed-scaffold agents, it treats the agent harness itself as the optimization target, applying beam-search-guided mutations conditioned on per-trajectory diagnostic reports. Across three benchmarks and four open-source LLMs, it improves the seed agent's pass@16 by 13.6% on average while using 17.5% fewer tokens.

Problem

Existing LLM-based cybersecurity agents operate with fixed, human-designed scaffolds that cannot adapt to the heterogeneous failure modes of diverse targets (CTF challenges, CVE exploitation, penetration testing). Generic self-evolution methods from other domains (ACE, HGM) fail because: (1) they permit unstructured scaffold rewrites with no localization; (2) they rely on clean test-suite feedback that is absent in adversarial cyber environments; (3) they follow a single-trajectory update path prone to error compounding and low diversity.

Method

CyberEvolver decomposes the agent scaffold \(A = (L_S, L_I, L_P, L_D)\) into four evolvable layers:

  • Strategy \(L_S\) (system prompt): reasoning framework and hypothesis-formation policy.
  • Environment Interface \(L_I\) (instance prompt): shell idioms, I/O patterns.
  • Perception \(L_P\) (observation layer): raw-output filtering and runtime feedback injection.
  • Domain Knowledge \(L_D\) (skill library): on-demand tactical playbooks.

Trajectory diagnosis converts raw rollout \(\tau\) into a compressed summary \(z\) (windowed 10-step summarization with selective verbatim retention and placeholder back-filling) and a structured diagnosis report \(d\) containing failure attribution, root-cause hypotheses, counterfactuals, and a progress score \(s \in [0, 100]\).

Beam search evolution (Algorithm 1): at each generation \(t\), the top-\(k\) agents by \(s\) are selected from population \(P_t\). From each parent, \(m\) child variants are sampled via four sequential LLM calls (one per layer, each building on prior edits in the same pass), validated syntactically, and rolled out. Default configuration: \(T=3\) generations, \(k=3\) beam width, \(m=3\) mutations per parent, yielding at most 16 rollouts per target.

Key Contributions

  • Identification of cybersecurity as a natural on-policy self-evolution domain, with characterization of three obstacles: unstructured mutation space, obscured mutation signal, low mutation diversity.
  • Four-layer evolvable scaffold architecture enabling localized, failure-attributed mutations instead of monolithic rewriting.
  • Exploit-semantic trajectory compression with windowed summarization, selective verbatim retention, and placeholder back-filling to recover actionable signal from adversarially noisy environments.
  • Beam search over agent variants with layer-wise LLM mutation calls, preventing single-path error accumulation.
  • Unified evaluation suite reorganizing three cybersecurity benchmarks (NYU-CTF, AutoPenBench, CVEBench) with large-scale parallel evaluation support.

Results

  • vs. seed agent pass@16: +13.6% average across all benchmarks and four backbone models (DeepSeek-V3.1, Kimi-K2.5, MiniMax-M2.5, Qwen3-235B), using 17.5% fewer tokens.
  • NYU-CTF: +8.9% to +15.2% over seed pass@16 depending on model.
  • AutoPenBench: +9.1% to +39.3% over seed pass@16.
  • CVEBench Zero-Day: +10.0% to +17.5% over seed pass@16.
  • CVEBench One-Day: +5.0% to +10.0% over seed pass@16.
  • Outperforms six human-designed agents (NYUCTFAgent, AutoPenBench-Agent, CyAgent, VulnBot, DCipher, T-Agent) and two adapted self-improvement methods (ACE, HGM).
  • On a 488-point blind SQLi challenge solved by 4.1% of 1,096 teams: seed agent fails; generation-3 agent solves in 18 steps via binary-search-guided blind injection.
  • Seed agent pass@k nearly saturates after \(k=4\) (+1.4% from pass@4 to pass@16); CyberEvolver continues improving past this ceiling.
  • HGM on CVEBench Zero-Day (Kimi-K2.5): 25.0% at 640 evaluations vs. CyberEvolver 37.5% at 16 nodes; 72% of HGM variants collapse to tool-wrapper mutations.

Limitations

  • Evaluation excludes vulnerability discovery benchmarks (deferred to future work).
  • Backbone models limited to four open-source frontier LLMs; proprietary model performance not assessed.
  • The progress score \(s\) is used only as a relative ranking signal within a generation, not as a calibrated distance-to-success metric.
  • On-policy evolution is target-specific; each new target requires fresh evolution, with no cross-task transfer.
  • Maximum 16 rollouts per target; compute budget analysis for harder targets is not explored in depth.
  • Paper text is truncated before full sibling diversity analysis results are presented.

Relevance to Harnesses / Meta-Harnesses

CyberEvolver is a direct instance of a meta-harness: the outer evolutionary loop (execute → summarize → diagnose → select → mutate) is itself a harness that automatically rewrites the inner agent scaffold (harness). The four-layer decomposition is a general template for structuring any agentic harness into independently mutable components, a design principle directly applicable to non-cybersecurity meta-harness work. The trace-to-diagnosis pipeline solves the universal harness-design problem of converting opaque execution feedback into structured mutation signals—a mechanism transferable to any harness that self-optimizes on task trajectories. The beam-search evolution strategy offers a concrete blueprint for diversity-preserving scaffold search, contrasting with the single-path update patterns that plague naive self-improving harness designs.