Fuzzwise: Intelligent Initial Corpus Generation for Fuzzing

🕒 Published (v1): 2025-12-24 22:17 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

FuzzWise is an LLM-based multi-agent framework for constructing high-quality initial corpora of seeds (ICS) for mutation-based greybox fuzzing. It merges the traditionally separate corpus generation and minimization phases into a single iterative loop by using a predictive coverage LLM (CodePilot) to evaluate each candidate seed before execution. On the FixEval benchmark, FuzzWise generates 42.1% fewer seeds while achieving 10.71% higher code coverage and 100% more runtime error detections than baselines.

Problem

Traditional ICS construction for coverage-guided fuzzing has two decoupled phases: (1) generate a large full corpus, then (2) minimize it via corpus distillation (e.g., AFL-cmin). This is wasteful because the minimization phase does not actively seek to increase coverage—it only removes redundant seeds—and execution-based coverage measurement is expensive, slow, or impossible in some environments. Random seed generation also fails to target error-triggering inputs, so even high-coverage corpora miss runtime bugs.

Method

FuzzWise orchestrates two LLM agents in a closed feedback loop:

1. Test Case Generator (TCG) — GPT-4: Generates a single candidate seed per iteration. Uses one of two prompts depending on current cumulative coverage: (a) coverage-increasing mode (marks covered vs. uncovered lines with > / ! annotations) or (b) error-detection mode (explicitly targets ArrayIndexOutOfBoundsException, NullPointerException, ArithmeticException, etc.) once 100% coverage is reached.
2. Coverage Predictor — CodePilot: An LLM-based static predictor that reasons about path constraints via step-by-step execution simulation to predict which lines a given input would cover, without actually running the program.

Each generated seed is immediately evaluated by CodePilot. If the predicted coverage union with the current corpus increases total coverage, the seed is added to ICS and cumulative coverage is updated; otherwise it is discarded and the TCG is re-prompted. Duplicates are pruned at the end. The loop terminates on a time limit or 100% predicted coverage.

Key Contributions

• FuzzWise algorithm: Unified, single-phase ICS construction that integrates generation and quality assessment, eliminating the generate-then-minimize pipeline.
• Tandem LLM agents: TCG (high recall, broad exploration) paired with CodePilot (high precision, coverage-guided filtering) to balance recall vs. precision.
• Execution-free coverage estimation: Uses CodePilot's static prediction to avoid environment setup and runtime overhead, enabling ICS construction in constrained or execution-hostile settings.
• Dual-mode prompting: Switches from coverage-maximizing to error-targeting generation once structural coverage saturates.
• Empirical evaluation on FixEval (50 Java + 50 Python programs, cyclomatic complexity 10–20) against JQF-AFL/Python-AFL baselines.

Results

• 42.1% fewer seeds generated vs. baselines while achieving 10.71% higher code coverage.
• 100% relative increase in runtime error detection compared to best baseline (Baseline 3) when ICS is used as fuzzer starting point (Java).
• Errors per Seed (EPS):
• Java: FuzzWise 0.167 vs. Baseline 3 0.052 (221% improvement).
• Python: FuzzWise 0.27 vs. Baseline 3 0.052 (>5× improvement).
• Errors per Time (EPT):
• Java: FuzzWise 0.008 vs. Baseline 3 0.003 (166.7% improvement).
• Python: FuzzWise 0.016 vs. Baseline 3 0.008 (2×).
• Baselines 1 and 2 (random seeds ± minimization) triggered zero errors across both Java and Python datasets.
• FuzzWise outperforms Baseline 3 on 76% of Java programs and 88% of Python programs for error detection.
• Underperforms Baseline 3 on 6 Java programs with cyclomatic complexity ≥18, attributed to insufficient time budget for complex programs.

Limitations

• Evaluated on a single benchmark (FixEval), which consists of competitive-programming submissions—these are relatively small, self-contained programs with integer/string I/O; generalization to real-world complex codebases is unverified.
• Restricted to programs within GPT-4's token context window, bounding applicable cyclomatic complexity to 10–20.
• Coverage predictor (CodePilot) is approximate; mispredictions can admit redundant seeds or reject useful ones, degrading ICS quality.
• Underperforms on high-complexity programs (cyclomatic complexity ~18) within the 5-minute budget, suggesting the approach is time-sensitive for harder targets.
• No evaluation of how ICS quality degrades when actual execution is unavailable versus when it is simply omitted for efficiency.
• Python coverage results show EPC parity (0.04) with Baseline 3 rather than improvement, indicating the coverage efficiency gain does not uniformly transfer across languages.

Relevance to Harnesses / Meta-Harnesses

FuzzWise is a textbook meta-harness pattern: it orchestrates two specialized LLM agents (generation + evaluation) in a feedback loop, dynamically routing work between them based on structured signals (predicted coverage delta), with an explicit termination condition and mode-switching logic—exactly the control flow skeleton that characterizes harness-level orchestration. The separation of a generator agent from a judge/filter agent with structured feedback is directly applicable to any meta-harness that needs to balance exploration (recall) against quality filtering (precision). The execution-free evaluation sub-agent (CodePilot) also demonstrates that harness inner loops need not be bottlenecked by heavyweight oracles—LLM-based predictive proxies can serve as fast gatekeepers.