Skip to content

Agentic Generation of AST Transformation Rules for Fixing Breaking Updates

🕒 Published (v1): 2026-06-23 11:27 UTC · Source: Arxiv · link

Why this paper was selected

Martin Monperrus (KTH, program repair); agentic workflow generates migration rules at scale

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

BIGBAG is an agentic framework that generates reusable AST transformation programs — rather than project-specific patches — to fix breaking Java dependency updates. A coding agent iterates through a generate-apply-verify loop against compiler feedback, producing a standalone executable that encodes the repair logic at the API level and transfers to any client broken by the same update. On 157 real-world breaking updates, the best configuration achieves a 94.3% compilable transformation rate and 78.6% fix rate, with 80%+ cross-project transferability for uniformly-used APIs.

Problem

Existing LLM-based breaking-update repair tools (e.g., Reyes et al., Fruntke & Krinke) generate project-specific patches that cannot be reused. When library \(L\) updates from \(v_i \to v_j\) and breaks \(N\) client projects, each must be repaired independently even though the root cause is identical across all clients. This limits scalability in real dependency ecosystems.

Method

BIGBAG orchestrates a coding agent through a four-step pipeline:

  1. Input context assembly — the agent receives: (a) the broken client project, (b) full Javadoc for the exact version of the chosen AST engine (Spoon or JavaParser; 1,500–1,900 HTML files per engine, navigated on-demand to avoid context saturation), (c) a minimal transformation template (empty Main.java + engine dependency in pom.xml), and (d) an api-spec.md listing all exported types of \(v_j\) plus the dependency Javadoc.
  2. Generate-apply-verify loop — the agent calls mvn compile, reads compiler errors, writes a standalone Java program that traverses the client AST, locates incompatible usages, and rewrites them. If the transformation fails to compile or throws at runtime, the error feeds back into the next generation attempt.
  3. Isolation verification — after the loop, BIGBAG re-applies the transformation to the original client in isolation (outside the agentic loop) to confirm the transformation alone — not any side-edits the agent may have made — resolves the failure.
  4. Cross-project transfer — verified transformations are applied to all other clients in the BUMP benchmark affected by the same \((L, v_i \to v_j)\) update; build success/failure is recorded per (transformation, target project) pair.

Invalid outputs are classified as Transformation Bypass (string matching instead of AST ops, pom.xml edits, or wrong library).

Key Contributions

  • BIGBAG framework: first approach to generate reusable AST transformations for fixing breaking Java dependency updates (open-source).
  • Empirical study: 8 configurations × 157 breaking updates (4 LLMs × 2 AST engines), measuring well-formedness (RQ1), fix rate (RQ2), and cross-project transferability (RQ3).
  • Engine impact characterization: AST engine choice is a first-order determinant of success — JavaParser vs. Spoon differ by up to 17.8% in compilable transformation rate.
  • Failure mode taxonomy: distinguishes transformation-not-generated vs. transformation-generated-but-incorrect; transfer failure is linked to usage non-uniformity across clients.

Results

  • Best configuration (Gemini-3.1-Pro + JavaParser): compilable transformation rate 94.3%, fix rate 78.6% on 157 BUMP breaking updates.
  • Engine gap: JavaParser generally outperforms Spoon; up to 17.8% difference in compilable transformation rate across the four models.
  • Cross-project fix rate: 33.3% overall; rises to ≥80% for breaking updates where all clients invoke the affected API element uniformly.
  • Dataset composition: 3 PATCH (1.9%), 63 MINOR (40.1%), 91 MAJOR (58.0%) updates; MAJOR updates have median 5 compilation errors, MINOR median 2.
  • Cost range: $60 (Qwen3-30B) to $2,600 (Gemini-3.1-Pro) per full experimental run.

Limitations

  • Cross-project transfer drops sharply (to 33.3% overall) when clients use the broken API element non-uniformly — the transformation encodes one usage pattern and misses others.
  • Each (model, engine) configuration is run once per breaking update; no repeated trials to measure variance.
  • Documentation is provided as on-demand file-system reads; the paper acknowledges this design assumption (agents may navigate inefficiently or miss relevant pages) as a threat to validity.
  • Evaluation is Java/Maven-only (BUMP benchmark); generalizability to other ecosystems (npm, PyPI) is untested.
  • Agent stopping condition is autonomous and may deliver malformed transformations, requiring the external verification step to catch them.
  • Text truncated before full RQ1–RQ3 result tables are shown; precise per-model/per-engine breakdowns are incomplete in the provided excerpt.

Relevance to Harnesses / Meta-Harnesses

BIGBAG is a domain-specific agentic harness that wraps a coding agent inside a deterministic four-step orchestration pipeline — context assembly, generate-apply-verify loop, isolation verification, and cross-project deployment — with explicit success/failure classification at each stage. The separation between the agentic loop (Step 2) and the external verification step (Step 3) is a canonical harness pattern: it prevents the agent's unconstrained side-effects from inflating reported fix rates, mirroring the verifier/judge role seen in broader meta-harness designs. The cross-project transfer step (Step 4) makes BIGBAG a scalable deployment harness: a single agent run produces an artifact that a harness then propagates across a fleet of targets, decoupling generation cost from per-project repair cost. For researchers building or studying meta-harnesses, this paper provides a concrete case study of how to structure agent autonomy, constrain output format (AST transformation vs. raw patch), and enforce isolation in verification.