Skip to content

QLCoder: A Query Synthesizer For Static Analysis of Security Vulnerabilities

🕒 Published (v1): 2025-11-11 17:06 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

QLCoder is an agentic framework that synthesizes CodeQL vulnerability-detection queries directly from CVE metadata by embedding an LLM in an iterative refinement loop with execution feedback. It exposes a custom MCP interface connecting a CodeQL Language Server (syntax guidance) and a ChromaDB RAG database (semantic retrieval), constraining the agent's reasoning to avoid common LLM failure modes. Built on Claude Code, it achieves a 53.4% query success rate on 176 real CVEs vs. 10% for bare Claude Code.

Problem

Writing static analysis queries (CodeQL, Semgrep, Infer) demands expertise in security, program analysis, and low-resource, evolving query DSLs. Existing query suites have poor coverage and precision. Naive LLM prompting produces syntactically invalid queries, hallucinates deprecated API constructs, and fails to capture inter-procedural taint propagation patterns that span multiple files and components.

Method

QLCoder runs a two-level loop: an outer refinement loop (up to \(N=10\) iterations) and an inner agentic conversation loop (up to \(M=50\) tool-call turns per iteration). Each outer iteration: 1. The LLM agent issues tool calls via a custom MCP interface to a ChromaDB RAG database (CWE definitions, sample CodeQL queries, API docs, CVE/patch diffs, AST snippets extracted from the target repo) and a CodeQL Language Server (completion, hover, diagnostics) to draft a candidate query \(Q_i\). 2. A CodeQL-based validator compiles \(Q_i\), executes it on both the vulnerable version \(P_{\text{vuln}}\) and patched version \(P_{\text{fixed}}\), and returns a structured feedback report: compilation status, match counts, recall/coverage, counterexample traces, and programmatically-generated next-step recommendations. 3. If \(\text{success}(Q_i; P_{\text{vuln}}, P_{\text{fixed}})\)—at least one path through the patched region on \(P_{\text{vuln}}\), zero such paths on \(P_{\text{fixed}}\)—the loop terminates; otherwise the agent uses the feedback to propose \(Q_{i+1}\).

Conversation history is reset per outer iteration to keep context compact. The MCP toolbox is deliberately small to prevent tool-selection confusion; full compile-and-run is deferred to iteration boundaries; web search is explicitly excluded after empirical degradation.

Key Contributions

  • QLCoder agentic framework: end-to-end CVE metadata → executable CodeQL path query, integrating execution-guided synthesis, structured RAG retrieval, and LSP-backed syntax validation.
  • Custom MCP interface: two MCP servers (ChromaDB and CodeQL Language Server) providing demand-driven, version-specific guidance without polluting the main LLM context.
  • Formal synthesis task definition: well-formedness, vulnerability detection (\(\exists \pi \in \llbracket Q \rrbracket(G_{\text{vuln}})\) s.t. \(\pi \cap \Delta V \neq \emptyset\)), and fix discrimination (\(\forall \pi \in \llbracket Q \rrbracket(G_{\text{fixed}}),\; \pi \cap \Delta V = \emptyset\)).
  • Empirical evaluation on CWE-Bench-Java: 176 CVEs, 111 Java projects (0.01–1.5 MLOC), 42 CWE categories, including 65 CVEs from 2025 targeting CodeQL 2.22.2.

Results

  • QLCoder: 100% compilation rate, 53.4% success rate, F1 = 0.7
  • Bare Claude Code: ~19% compilation, 10% success (or 0% by Gemini CLI's baseline numbers)
  • Gemini CLI (best agentic baseline): 19% compilation, 0% success
  • IRIS (LLM-assisted static analyzer): F1 = 0.048
  • Default CodeQL query suite: F1 = 0.073

Limitations

  • Evaluated only on Java projects (CWE-Bench-Java); generalization to other languages (Python, C/C++) is unvalidated.
  • Patch diffs require manual filtering to isolate CVE-relevant files before populating the RAG database; this step is not automated.
  • The success criterion (\(\text{success}(\cdot)\)) permits false-positive paths in both versions; precision is left as a soft property rather than a hard constraint.
  • Outer iteration budget (\(N=10\)) and inner turn budget (\(M=50\)) are fixed heuristics; the loop may fail for CVEs requiring deeper exploration.
  • The text is truncated; a dedicated limitations section may contain additional caveats.

Relevance to Harnesses / Meta-Harnesses

QLCoder is a textbook harness architecture: a structured outer loop drives an inner agentic loop, with a validator providing typed feedback that flows back into the next iteration's prompt—exactly the "synthesis loop with execution feedback" pattern central to harness design. The paper's Context Engineering section directly addresses the harness designer's core problem: what information to expose to the agent, when, and in what form, explicitly finding that unrestricted tool access (compile-and-run on demand, web search) degrades performance and that a small, scoped MCP toolbox outperforms a large heterogeneous one. The decision to reset conversation history at each outer iteration to prevent context bloat is a concrete harness-level engineering choice. For researchers tracking harnesses/meta-harnesses, QLCoder is a strong case study in MCP-based context engineering, outer-loop/inner-loop decomposition, and deliberate constraint of agentic tool surfaces.