Project Prometheus: Bridging the Intent Gap in Agentic Program Repair via Reverse-Engineered Executable Specifications¶

🕒 Published (v1): 2026-04-19 14:27 UTC · Source: Arxiv · link

Why this paper was selected

Reverse-engineered executable specs bridge intent gap in agentic program repair

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Prometheus is a multi-agent Automated Program Repair (APR) framework that inverts the standard repair workflow: instead of asking an LLM to fix a bug directly, it first reverse-engineers executable Behavior-Driven Development (BDD/Gherkin) specifications from runtime failure reports, validates them against ground-truth code via a Requirement Quality Assurance (RQA) Loop, then supplies these verified specs to a separate coding agent. On 680 Defects4J bugs, Prometheus achieves a 93.97% correct patch rate and a 74.4% rescue rate on bugs a strong blind agent could not solve.

Problem¶

Agentic APR systems suffer from the "Intent Gap": patches satisfy failing tests but diverge from developer intent because current approaches use vague proxies (natural language summaries, ad-hoc adversarial test cases) that lack contractual precision. Without deterministic behavioral constraints, powerful coding agents exhibit "Berserker-style" behavior—structurally invasive rewrites that mirror the broad scope of their underspecified inputs rather than making surgical, semantically bounded corrections.

Method¶

Prometheus decomposes repair into three specialized agents in a sequential pipeline:

Architect (Gemini-3.0-Pro) — Specification Inference. Given the failing test source and production code, the Architect performs two-stage Chain-of-Thought reasoning: (a) root cause analysis of the runtime failure, then (b) synthesis of a Gherkin .feature file (Given-When-Then scenarios) encoding the intended behavior as an executable contract.
Engineer — RQA Loop (Sandwich Verification). The Engineer validates the candidate specification \(S\) bi-directionally: \(S\) must fail on \(C_{\text{buggy}}\) (negative verification) and pass on \(C_{\text{fixed}}\) (positive verification). Only specifications satisfying both conditions propagate. In experimental settings, \(C_{\text{fixed}}\) from Defects4J metadata serves as the proxy oracle; the paper argues production deployment would substitute human review of Gherkin scenarios.
Fixer (Qwen-3.0-Coder) — Enlightened Repair. The Fixer receives the validated spec \(S\) in addition to the failing test and source code, and is constrained to modify only the single primary suspicious file (from Defects4J metadata) to isolate the effect of specification guidance from fault localization.

The composite evaluation strategy: Blind Fixer runs on all 680 bugs; Enlightened Fixer is applied only to the 160 bugs the Blind Fixer failed, and total fixes are summed (520 + 119 = 639).

Key Contributions¶

Prometheus framework: first APR system that treats specification inference as the primary task, using BDD reverse-engineering to generate executable intent contracts before code repair.
RQA Loop: "Sandwich Verification" mechanism using ground-truth developer fixes as proxy oracles to validate inferred specifications, isolating spec quality as an independent variable.
Intent-Behavior Mirroring Effect: empirical observation that the structural invasiveness of generated patches mirrors the structural scope of the input requirement—precise specs yield surgical repairs.
Quantitative results: 93.97% correct patch rate (639/680) on Defects4J v3.0.1, with a 74.4% rescue rate (119/160) on hard bugs, and 4.4× advantage over SOTA on complex defects.

Results¶

Overall: 639/680 correct patches (93.97%); Blind Fixer alone: 520/680 (76.5%).
Rescue Rate: 119/160 hard bugs rescued by Enlightened Fixer (74.4%).
SOTA comparison on hard 160 bugs: TSAPR: 22; RepairAgent: 27; Prometheus: 119 (≈4.4× gap).
Project highlights: 100% rate on Cli, JacksonCore, Chart, Mockito, Gson, JxPath; Math improved from 48/106 blind to 87/106 total (82.1%); Compress from 29/47 blind to 43/47 (91.5%).
Cost breakdown (per defect): Architect: 136.77 s, 372K tokens (~6.4% budget); Engineer: 671.67 s, 3.37M tokens (~58.2%); Fixer: 452.64 s, 2.04M tokens (~35.4%); total pipeline: ~1,261 s, ~5.78M tokens.
Qualitative cases: In JxPath-6, the BDD-guided patch surpassed the ground-truth developer fix (handled arrays in addition to Collections); in Mockito-5, the spec prevented catastrophic catch-block deletion; in Chart-7, BDD blocked over-simplification that introduced regression.

Limitations¶

Oracle dependency: The RQA Loop requires \(C_{\text{fixed}}\) as a proxy oracle, restricting the current setup to benchmark evaluation settings; production deployment requires substituting human Gherkin review.
Single-file constraint: The Fixer is restricted to one primary suspicious file, causing failures on multi-file bugs (e.g., Time-1) and inducing "Cognitive Dissonance Hallucination" when root cause spans files.
Closure exclusion: The largest Defects4J project (Closure) was excluded due to legacy build system incompatibility and excessive RQA Loop latency, limiting coverage.
Engineer cost: Environment setup (compiling legacy Java projects, executing test harnesses multiple times) dominates cost at 58.2% of tokens and ~672 s/defect—the "Digital Archaeology" bottleneck.
Composite evaluation design: The reported 93.97% conflates blind and enlightened fixer runs; the Enlightened Fixer was never tested on bugs the Blind Fixer already solved, leaving potential degradation scenarios unexplored.

Relevance to Harnesses / Meta-Harnesses¶

Prometheus is a canonical example of a meta-harness pattern: it wraps an inner coding agent (Fixer) inside an outer harness (Architect + RQA Loop) whose job is to generate, validate, and supply executable specifications that constrain the inner agent's action space. The RQA Loop is itself a harness-level quality gate implementing a "pass/fail probe on two environments" pattern that is directly generalizable to any harness needing to validate intermediate artifacts before passing them downstream. The Sandwich Verification mechanism—forcing a candidate artifact to exhibit failure in one environment and success in another before promotion—is a reusable harness design primitive applicable beyond APR. For researchers building meta-harnesses that orchestrate LLM agents, Prometheus demonstrates that investing harness budget in specification-quality validation rather than patch-quantity sampling yields disproportionate returns on hard tasks.