Skip to content

EvoRepair: Enhancing Vulnerability Repair Agents Through Experience-Based Self-Evolution

🕒 Published (v1): 2026-05-28 15:46 UTC · Source: Arxiv · link

Why this paper was selected

EvoRepair: cross-vulnerability experience accumulation for self-evolving repair agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

EvoRepair is a self-evolving agent framework for automated vulnerability repair (AVR) that maintains a persistent, quality-scored experience bank, enabling LLMs to accumulate intra-vulnerability knowledge within a session and reuse cross-vulnerability patterns across sessions. It achieves 90.46% overall repair rate on PATCHEVAL + SEC-bench with GPT-5-mini, outperforming the strongest LLM-based baseline (LoopRepair) by 33–39 percentage points.

Problem

LLM-based AVR agents treat each vulnerability as an isolated task, discarding all intermediate repair signals after each attempt. This causes two compounding failures: (1) intra-vulnerability: repeated unproductive paths within a single repair session because no distilled knowledge carries across attempts; (2) cross-vulnerability: identical mistake patterns recur across different CVEs because no mechanism reuses prior repair trajectories. Simply increasing repair steps yields diminishing returns ("blind scaling") under fixed budgets.

Method

EvoRepair wraps a base ("vanilla") ReAct agent in a cyclic learn-and-repair meta-loop:

  1. Domain-specific experience schema — Each experience is a structured 5-dimension record: (A) vulnerability introduction/analysis, (B) repair strategy, (C) trajectory analysis (positive/negative action patterns), (D) actionable experience summary with applicability preconditions and concrete guidance, (E) reflection & improvement for follow-up.

  2. Experience retrieval — Before each repair turn, candidates are drawn from the bank by similarity: $\(C = \text{Top}_M\{\text{sim}(q, e_i)\}_{i=1}^N\)$ then reranked by a composite score blending similarity and accumulated experience quality \(s_{\text{exp}}\): $\(s'(e) = \mu \cdot \text{sim}(q, e) + (1-\mu) \cdot s_{\text{exp}}(e)\)$ and the top-\(K\) are selected. Retrieved experiences are injected into the agent's context via a \(2\times2\) memory matrix (outcome: success/fail \(\times\) source: self/other).

  3. Vanilla agent repair — Minimal Bash toolkit + task-specific skill prompts (understand, reproduce PoC, localize, verify patch, reset environment). Operates within bounded step/cost budgets.

  4. Experience construction — After each repair attempt, the trajectory is compressed and distilled into a new structured experience. Historical experiences retrieved during the turn are also refined by reflecting on what was correct or erroneous.

  5. Experience scoring (LLM-as-a-Judge) — Each synthesized experience receives a combined score \(s_{\text{exp}} = \lambda \cdot \text{quality} + (1-\lambda) \cdot \text{generalizability}\) (\(\lambda=0.5\)); the bank is updated accordingly.

  6. Cost-aware early stopping — The turn-level yield rate \(\alpha = \beta/\gamma\) (ratio of repair-performance growth rate to cost growth rate) triggers termination when marginal gain per token falls below a threshold.

Cold-start is handled via two warm-up strategies: (a) injecting official patches from similar historical CVEs, or (b) pre-populating the bank with pre-repair experiences from earlier-disclosed vulnerabilities, excluding the target CVE itself.

Key Contributions

  • First experience-based self-evolving framework for AVR, introducing intra- and cross-vulnerability knowledge reuse as a research dimension.
  • Standardized 5-dimension domain-specific experience schema enabling retrievable, refinable, transferable repair knowledge.
  • Quality-aware experience bank with composite similarity + LLM-judge scoring and context-bounded compression.
  • Yield-rate-based early stopping to prevent runaway cost under fixed budgets.
  • Demonstrated cross-dataset, cross-language (Java/C/JavaScript), and cross-model transfer via experience bank portability.

Results

  • PATCHEVAL: 93.47%; SEC-bench: 87.00%; Overall: 90.46% (GPT-5-mini backbone, 12 baselines).
  • Outperforms LoopRepair (strongest LLM-based baseline) by +39.56% (PATCHEVAL) and +33.50% (SEC-bench).
  • Outperforms IntentFix by +70.86% (PATCHEVAL) and +50.50% (SEC-bench).
  • Outperforms the recent self-evolving SE agent Live-SWE-Agent by +6.98% overall.
  • Transfer experiments on VUL4J confirm cross-dataset, cross-language, and cross-backbone generalization.

Limitations

  • Evaluated with GPT-5-mini as primary backbone; other model comparisons limited to transfer ablations.
  • Cold-start warm-up assumes access to historical CVE data or pre-run repairs; both introduce potential data leakage risks.
  • Experience scoring depends on LLM-as-a-Judge reliability; systematic biases in the judge propagate to bank quality.
  • Context-window compression during experience construction may discard edge-case signals critical for rare CWE types.
  • No analysis of failure modes where retrieved cross-vulnerability experiences mislead repair (negative transfer).

Relevance to Harnesses / Meta-Harnesses

EvoRepair is structurally a meta-harness: it wraps a base agent (the vanilla ReAct loop) with an outer orchestration layer that manages state (experience bank), retrieval, quality evaluation, and adaptive termination across multiple agent invocations — a classic harness pattern. The cyclic learn-and-repair paradigm directly mirrors meta-harness designs that run agents in iterations, accumulate structured output between runs, and modify agent context for the next run. The LLM-as-a-Judge scoring + quality-aware bank update is an instance of the meta-harness evaluator pattern, where the harness filters and ranks agent outputs before they propagate forward. For researchers tracking harnesses, this paper operationalizes two underexplored harness responsibilities: cross-run state management (experience bank) and cost-aware loop termination (yield-rate early stopping).