Revelio: Cost-Efficient Agentic Memory Safety Vulnerability Detection For Repository-Scale Codebases

🕒 Published (v1): 2026-06-20 23:17 UTC · Source: Arxiv · link

Why this paper was selected

Cost-efficient multi-agent harness for repo-scale memory-safety detection; addresses scalability directly

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Revelio is a two-stage agentic framework for memory-safety vulnerability detection in repository-scale C/C++ codebases. It pairs cheap LLMs for high-recall hypothesis generation with stronger models plus sanitizer-backed confirmation to achieve zero false positives. On 100 CyberGym/Arvo projects it finds 175 vulnerabilities vs. 55 for Claude Code (Opus 4.7), at comparable token cost, and discovers 19 zero-days in OSS-Fuzz-hardened projects for ~$300 total.

Problem

LLM-based vulnerability detection on real codebases suffers from three compounding failures: (1) hallucinated findings (high false-positive rate), (2) poor recall on subtle bugs requiring reasoning about rare execution paths, and (3) prohibitive cost when frontier models are applied uniformly across every file in a large repository. Fuzzing alone misses logic-dependent or parser-state bugs; static analysis lacks semantic reasoning. No prior method achieves simultaneously high recall, zero false positives, and deployable cost at repository scale.

Method

Revelio decomposes detection into two qualitatively distinct stages:

Stage 1 — Hypothesis Generation (breadth, cheap model): For every C/C++ source file, a sub-agent performs multi-pass LLM analysis (whole-file summary, feature interaction, unchecked-argument, per-function passes) to emit structured JSON hypotheses (see Table 1: hypo_id, primitive, severity, sanitizers, harnesses, etc.). Lightweight tree-sitter-based static preprocessing identifies unchecked function parameters as hints. Raw hypotheses are then triaged (scope filter, attacker-reachability check), deduplicated via line-range overlap + CWE label + LLM pairwise comparison, and ranked by reachability × severity × confidence. Candidate test harnesses from the target codebase are matched via symbol-table lookup.

Stage 2 — Hypothesis Confirmation via PoV Construction (depth, stronger model): For each ranked hypothesis, an agent receives a structured context packet and three tools: shell (repository inspection + file writing), validate (runs target binary under ASan/MSan/UBSan in Docker, returns sanitizer output), and finish (structured result submission). The agent writes a Python script generating raw PoV bytes, then iteratively refines based on sanitizer feedback. A hypothesis is confirmed only when an independent re-execution in a fresh subprocess triggers a sanitizer-detected crash — the agent cannot self-report success. A reporter sub-agent then produces a developer-facing report with reproduction steps.

The model tier split is: Haiku 4.5 for Stage 1 scanning, Sonnet 4.6 for Stage 2 PoV construction.

Key Contributions

• Formalizes memory-safety detection as a two-stage pipeline separating speculative code review (recall-optimized, cheap model) from executable confirmation (precision-optimized, deterministic sanitizer oracle), achieving zero false positives.
• Designs and implements Revelio in 4.1 K lines of Python with a DefaultAgent abstraction, YAML/Jinja prompt templates, and Docker-isolated execution environments.
• Discovers 19 previously unknown memory-safety vulnerabilities (7 CVEs) in projects fuzzed continuously for 5–8 years by OSS-Fuzz, at ~$42/project median cost.
• Outperforms frontier coding agents (Claude Code Opus 4.7, Codex GPT 5.5, Sorcar Opus 4.7) on CyberGym/Arvo benchmark: 175 vs. 55/39/31 vulnerabilities found, with zero false positives vs. substantial FP rates for baselines.

Results

• Zero-day discovery: 19 previously unknown vulnerabilities (7 CVEs) across 7 OSS-Fuzz projects (5–8 years of continuous fuzzing); median ~1 hr/project, ~$42/project, $300 total.
• CyberGym/Arvo (100 projects): Revelio (Haiku 4.5 → Sonnet 4.6) finds 175 vulnerabilities at $9.10/vuln vs. Claude Code Opus 4.7: 55 at $16.20/vuln; Codex GPT 5.5: 39 at $8.96/vuln; Sorcar Opus 4.7: 31 at $6.27/vuln.
• Revelio has zero false positives; all baselines have substantial FP rates.
• Revelio does not require non-public models (e.g., Anthropic Mythos Preview) to achieve these results.

Limitations

• Evaluated only on C/C++ codebases; generalization to memory-safe languages or other vulnerability classes is not addressed.
• Requires pre-existing test harnesses in the target codebase for PoV construction; projects without harnesses cannot be confirmed.
• Stage 2 confirmation is bounded by the agent's ability to synthesize valid input formats — complex parsers or deeply constrained input grammars may defeat PoV construction even when a real bug exists (false negatives at Stage 2).
• Paper text is truncated before full ablation (RQ3) results are presented; quantitative contribution of individual components (static preprocessing, multi-pass generation, deduplication) is not fully reported here.
• Docker execution adds infrastructure overhead and is not evaluated for portability across build systems.

Relevance to Harnesses / Meta-Harnesses

Revelio is a concrete instantiation of a heterogeneous multi-agent meta-harness: a top-level orchestrator dynamically instantiates typed sub-agents (DefaultAgent) from YAML/Jinja templates, routes artifacts (hypotheses → context packets → PoV scripts → sanitizer reports) between stages, and enforces a strict evidence boundary via external deterministic tools rather than model self-assessment. The design pattern — cheap broad agents feeding a ranked queue consumed by expensive narrow agents, with non-LLM oracles as gatekeepers — is directly applicable to any meta-harness that must balance coverage against cost. The explicit schema for inter-stage artifact hand-off (Table 1/2) and the independent re-execution guard against agent self-reporting are both transferable architectural primitives for harness designers.