Skip to content

Detecting Vulnerability-Inducing Commits via Multi-Stage Reasoning with LLM-Based Agents

🕒 Published (v1): 2026-07-07 02:54 UTC · Source: Arxiv · link

Why this paper was selected

Multi-stage LLM agent reasoning for security-critical commit analysis — practical agentic pipeline

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VIC-RAGENT is a multi-agent LLM framework for just-in-time vulnerability detection that decomposes commit analysis across role-specialized agents (Code Analyst, Target Analyst, Vulnerability Inspector, Document Specialist, Audit Supervisor) operating in a structured four-stage pipeline. The framework integrates RAG over a self-updating vulnerability knowledge base to enable case-based reasoning. Across three LLMs and a real-world dataset, it achieves 1.2–1.7× higher F1-scores than the strongest baseline (CodeAgent), driven primarily by substantially improved recall.

Problem

Existing just-in-time vulnerability detection (JIT-VD) methods—feature-based, deep learning, and single-agent LLM approaches—fail to jointly reason over heterogeneous commit signals (code diffs, commit messages, file-level context). Direct prompting and CoT achieve high precision but catastrophically low recall (as low as 6%), while agent frameworks like CodeAgent lack multi-stage verification mechanisms, causing uncontrolled false positives and missed vulnerabilities.

Method

VIC-RAGENT structures commit analysis as a four-stage pipeline with distinct agent roles and temperature settings per stage:

  • Stage 0 – Preparation: A Code Analyst (CA, \(T=0.2\)) extracts structural relationships (simplified call graphs, data-flow patterns) from the diff+context; a Target Analyst (TA, \(T=0.2\)) categorizes commit intent (bug fix, feature addition, refactoring, etc.) and flags goal–implementation inconsistencies.
  • Stage 1 – Preliminary Inspection (high-recall candidate generation): A Vulnerability Inspector (VI, \(T=0.4\)) screens modified fragments against six vulnerability categories (I/O Validation, Memory Safety, Web Security, Authentication & Authorization, Resource Management, File/Path Handling), deliberately relaxing verification constraints to maximize recall.
  • Stage 2 – Reanalysis (type-guided verification): VI re-evaluates each candidate under its predicted vulnerability type. A RAG lookup retrieves the most similar historical case from the knowledge base (CodeBERT embeddings, cosine similarity \(\geq 0.85\)) and injects it as additional evidence; if no similar case exists, context-only reasoning proceeds.
  • Stage 3 – Final Decision (conservative aggregation): VI operates at \(T=0.1\) to produce deterministic fragment-level verdicts; a commit is labeled VIC if any fragment is confirmed. An Audit Supervisor (AS, \(T=0.1\)) validates agent outputs for consistency. If a VIC is confirmed, a Document Specialist (DS, \(T=0.3\)) generates a structured security report added to the knowledge base; FP-confirmed entries are periodically pruned.

The knowledge base uses a two-level structure (category → security reports) indexed by CodeBERT embeddings for efficient similarity retrieval.

Key Contributions

  • Multi-agent architecture with role-specialized agents (CA, TA, VI, DS, AS) providing complementary structural and semantic views of commits.
  • Four-stage coarse-to-fine reasoning pipeline (preparation → preliminary inspection → reanalysis → final decision) with per-stage temperature calibration.
  • Self-expanding vulnerability knowledge base with RAG-based retrieval and FP-pruning to prevent error propagation.
  • Empirical evaluation across three LLMs (DeepSeek-V3.2, GPT-4o-mini, Qwen-Plus) on V-SZZ dataset and a post-cutoff CVE generalization set.

Results

  • Main benchmark (V-SZZ, 241 commits, 106 VIC / 135 VFC):
  • DeepSeek-V3.2: VIC-RAGENT F1 = 57% vs. CodeAgent 33% (1.7×); recall 48% vs. 22%.
  • GPT-4o-mini: F1 = 60% vs. CodeAgent 49% (1.2×); recall 75% vs. 60%.
  • Qwen-Plus: F1 = 66% vs. CodeAgent 55% (1.2×); recall 58% vs. 58%, precision 76% vs. 52%.
  • CoT with DeepSeek-V3.2: precision 86% but recall 6% — illustrative of baseline pathology.
  • Ablation (DeepSeek-V3.2): removing Stage 2 drops F1 from 57% → 54% (TP 51→46, FN 55→60); removing Stage 3 raises recall but increases FP from 22 → 30 and reduces TN from 113 → 105.
  • Generalization (20 commits from 11 CVEs, majority post-Nov 2025 cutoff): VIC-RAGENT F1 = 60%, recall = 55% vs. CodeAgent F1 = 53%, recall = 45%.
  • Cost (DeepSeek-V3.2): $0.0487/commit for VIC-RAGENT vs. $0.0217 for CodeAgent; 103 s/commit vs. 80 s.

Limitations

  • Dataset is small (241 commits post-filtering from 360), limiting statistical power and generalization claims.
  • All agent roles use the same underlying LLM per experiment; no cross-model role specialization is explored.
  • Knowledge base effectiveness is limited at initialization (cold-start) and degrades if FP pruning is infrequent or inaccurate.
  • \(\sim2\times\) cost and \(\sim1.3\times\) latency over CodeAgent may be prohibitive for high-throughput CI pipelines without early-exit optimizations.
  • Embedding similarity threshold (0.85) for RAG retrieval is fixed and not ablated.
  • Generalization set is only 20 commits, making recall/F1 estimates on recent CVEs noisy.

Relevance to Harnesses / Meta-Harnesses

VIC-RAGENT is a domain-specific multi-agent harness: it encodes a fixed orchestration graph (CA → TA → VI×3 → DS/AS) with per-stage temperature policies, structured JSON inter-agent communication, and a RAG knowledge store as a persistent side-channel — all architectural patterns central to meta-harness design. The Audit Supervisor role directly instantiates the "meta-harness oversight" pattern, where one agent monitors and validates the outputs of subordinate agents rather than performing domain reasoning itself. The coarse-to-fine staged pipeline (high-recall generation → type-guided filtering → conservative aggregation) is a reusable harness template applicable beyond vulnerability detection to any multi-hypothesis verification task. For researchers tracking harnesses, this paper provides a concrete, ablated case study of how stage sequencing, role decomposition, and temperature scheduling interact to shape precision/recall trade-offs.