Skip to content

Security Is Relative: Training-Free Vulnerability Detection via Multi-Agent Behavioral Contract Synthesis

🕒 Published (v1): 2026-04-21 03:02 UTC · Source: Arxiv · link

Why this paper was selected

Multi-agent behavioral contract synthesis for vulnerability detection; deduplication-robust

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Phoenix is a training-free, three-agent pipeline for vulnerability detection that resolves the "semantic ambiguity problem" by synthesizing Gherkin behavioral specifications as a structured intermediate representation between code analysis and binary classification. On the PrimeVul Paired Test Set it achieves F1 = 0.825 and Pair-Correct = 64.4%, surpassing prior state-of-the-art by large margins using open-source models 48× smaller than competing baselines.

Problem

Deep learning vulnerability detectors trained on legacy benchmarks (Big-Vul) catastrophically degrade under rigorous evaluation: models achieving F1 > 0.68 collapse to as low as 0.031 on PrimeVul due to dataset leakage and label noise. The root cause is the semantic ambiguity problem: identical code patterns can be secure or vulnerable depending on project-specific sanitizers, architectural conventions, and runtime context. Over 90% of real-world vulnerabilities depend on context beyond the function boundary, and PrimeVul itself contains "Double Standard" pairs where code with >99% sequence similarity receives opposite labels across different CVEs (e.g., gen_assignment() in mruby labeled simultaneously good and bad, both CWE-125). Any global-classification approach is therefore fundamentally inadequate.

Method

Phoenix decomposes zero-shot vulnerability detection into three sequential specialized agents, each operating via structured XML-delimited prompts at temperature 0.2:

  1. Agent 1 — Semantic Slicer: Given a paired (bad, good) function plus CVE description and commit message, it identifies the diff, consults the oracle, then surgically prunes boilerplate, replacing heavily cut sections with placeholders. Average reduction: 5,298 → 911 characters (82.8%).

  2. Agent 2 — Requirement Reverse Engineer: Given the sliced pair, it reverse-engineers the implicit security contract as a Gherkin .feature specification in Given-When-Then format. Constraints enforce exactness (precise boundary conditions, not vague descriptions), minimality, and that scenarios describe secure behavior only. This Gherkin spec is the key intermediate representation — an explicit, structured encoding of what distinguishes the patch from the vulnerability.

  3. Agent 3 — Contract Judge: Given the Gherkin spec and a single target function (vulnerable or patched, evaluated independently), it performs a closed-form compliance check: does this code satisfy the behavioral contract? Detection is thus transformed from open-ended pattern matching ("Is this vulnerable?") to formal verification ("Does this satisfy the spec?").

All three agents include a <THINKING> section capped at 4 sentences. Agent 1 is fixed as Qwen3.5-9B; Agent 2 and Agent 3 are varied across 5 model families (7–14B) in a cross-model ablation.

Key Contributions

  • Phoenix achieves F1 = 0.825, P-C = 64.4% on PrimeVul Paired, vs. RASM-Vul (F1 = 0.668, P-C = 21.4%) using DeepSeek-V3 671B and VulTrial (F1 = 0.563, P-C = 18.6%) using GPT-4o — with models 48× smaller.
  • First system to automatically reverse-engineer strict Gherkin behavioral specifications from vulnerability-fix pairs and enforce them as zero-shot compliance criteria.
  • 25-configuration ablation across 5 model families quantifying each agent's contribution and the cross-model sensitivity of specification quality.
  • Empirical demonstration that security is a relative property: 18% (17/97) of Phoenix's false positives identify genuine security concerns in developer-patched code, suggesting PrimeVul labels are noisy.

Results

  • F1 = 0.825, P = 0.792, R = 0.860, P-C = 64.4% (optimal: A2 = Qwen2.5-Coder-14B, A3 = Qwen3.5-9B) on PrimeVul Paired Test Set (859 samples / 427 valid pairs)
  • Baselines on same benchmark:
  • RASM-Vul (RAG, DeepSeek-V3 671B): F1 = 0.668, P-C = 21.4%
  • VulTrial (adversarial debate, GPT-4o): F1 = 0.563, P-C = 18.6%
  • QLCoder (CodeQL synthesis, Java): F1 = 0.70 (different benchmark)
  • Ablation (Gherkin is decisive):
  • RAW monolithic zero-shot → Feature (full pipeline): +0.092 to +0.351 F1 across models
  • Blind (sliced only, no Gherkin): +0.00 to +0.06 over RAW — minimal gain
  • Gherkin stage alone accounts for +0.09 to +0.29 over the sliced baseline
  • Specification quality matters: Qwen2.5-Coder-14B as Agent 2 dominates all Agent 3 pairings (avg. F1 = 0.789 vs. Llama-3.1-8B avg. 0.728); code-specialized models produce higher-fidelity contracts
  • Agent 1 failure rate: 0.9% (8/435 pairs dropped due to XML formatting errors)

Limitations

  • Evaluated exclusively on C/C++ (PrimeVul); generalization to other languages untested.
  • Requires paired (vulnerable + fixed) code at inference time — the Gherkin spec is synthesized from the pair, so detecting vulnerabilities without a known patch is not directly supported by the described pipeline.
  • 0.9% XML formatting failure rate from Agent 1; structured output reliability is a ceiling on scalability.
  • Evaluation on a single benchmark (PrimeVul Paired Test Set, 435 pairs); PrimeVul's own label quality is questionable, as the authors demonstrate 18% of "false positives" are arguably true positives.
  • No runtime or cost analysis; three sequential LLM calls per sample pair with large context may be expensive at scale.
  • Agent 1 is not ablated across models — fixed as Qwen3.5-9B throughout.

Relevance to Harnesses / Meta-Harnesses

Phoenix is a canonical example of a sequential multi-agent harness where the pipeline architecture itself is the primary engineering contribution: each stage is a specialized sub-agent with a distinct prompt schema, and the harness introduces a structured intermediate representation (Gherkin specifications) that transforms the task type across stages — from code analysis to contract synthesis to compliance verification. The explicit use of XML-delimited I/O schemas between agents is a concretely described inter-agent communication protocol, directly relevant to designing robust meta-harness interfaces. The ablation methodology (25 configurations, cross-model matrices varying each agent independently) offers a replicable template for evaluating agent contribution within a multi-stage harness. The prior system Project Prometheus, also from the same authors, used an architecturally identical three-agent harness for Automated Program Repair, suggesting this Slicer→Synthesizer→Judge pattern is generalizing as a harness design pattern across domains.