Skip to content

ReProAgent: Tool-Augmented Multi-Stage Agentic Generation of Bug Reproduction Tests from Issue Reports

🕒 Published (v1): 2026-07-10 06:25 UTC · Source: Arxiv · link

Why this paper was selected

Multi-stage agentic pipeline with tool augmentation for bug reproduction; concrete harness recipe

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ReProAgent is a four-stage agentic framework that generates bug reproduction tests from natural-language issue reports by decomposing the task into bug localization, root cause analysis, test planning, and test generation. It equips each stage with a dedicated toolset—sequential thinking, hybrid code retrieval (keyword + knowledge graph), and runtime sandbox execution—rather than relying on a fixed prompt pipeline. On SWT-bench-lite and SWT-bench-verified it achieves 58.43% and 70.30% reproduction rates, outperforming all baselines at an average cost of $0.14 per instance.

Problem

Existing LLM-based reproduction test generators (LIBRO, Issue2Test, AssertFlip) follow fixed prompt pipelines that (1) rely solely on text-based retrieval and thus miss cross-file behavioral dependencies in repository-scale codebases, and (2) treat generation as a single step without explicitly modeling the intermediate planning stages a human developer actually performs (localization → root cause → test design → validation).

Method

ReProAgent organizes generation as a four-stage agentic loop:

  1. Bug localization via hierarchical analysis — the agent identifies suspicious files, classes, and functions using keyword-based retrieval (grep/glob/read-file) and a Tree-sitter-constructed code knowledge graph that encodes cross-file class/call dependencies.
  2. Root cause analysis via execution path — the agent traces call chains and dependency paths through the knowledge graph and interactive Python execution to infer the bug-triggering execution path.
  3. Assertion-aware test planning — the agent inspects existing tests and project conventions to derive setup patterns, invocation strategies, and expected failure conditions.
  4. Test generation via triadic review — candidate tests are executed inside a Docker sandbox; a triadic review (three independent LLM evaluators) assesses semantic correctness, and the agent iteratively refines based on execution feedback and review signals.

Across all stages, a sequential-thinking tool acts as a high-level controller, maintaining an explicit step-indexed reasoning trajectory with control signals for continue / revise / branch, forming a closed think–act–observe–revise loop.

Key Contributions

  • A four-stage multi-stage agent framework (ReProAgent) purpose-built for repository-level issue reproduction test generation.
  • A task-specific toolset combining sequential-thinking for reflection, hybrid retrieval (command-line + knowledge-graph), and Docker-sandboxed runtime interaction tools.
  • Empirical evaluation on SWT-bench-lite and SWT-bench-verified against multiple baselines and LLM backbones, plus ablation studies of individual components.
  • Demonstration that integrating ReProAgent's reproduction tests with SWE-agent raises the number of successfully resolved issues from 143 to 153 on SWT-bench-lite.

Results

  • SWT-bench-lite: 58.43% reproduction rate (vs. AssertFlip at ~38%, a +53.76 pp relative improvement stated in text as improving by 53.76%).
  • SWT-bench-verified: 70.30% reproduction rate (vs. AssertFlip improved by 54.51%).
  • vs. OpenHands (same GPT-5-mini backbone): +20.43 pp on SWT-bench-lite, +7.90 pp on SWT-bench-verified.
  • Cross-backbone generalization: 64.37% (GPT-5-mini), 58.47% (Qwen3-Coder), 52.11% (DeepSeek-V3.2), 50.43% (GLM-4.6).
  • Downstream repair: SWE-agent resolved issues increased from 143 → 153 when integrated with ReProAgent's tests on SWT-bench-lite.
  • Cost: $0.14 per instance on average.

Limitations

  • Evaluation restricted to Python repositories (SWT-bench); cross-language generalizability is not demonstrated.
  • Knowledge graph construction via Tree-sitter adds preprocessing overhead not fully characterized in the paper.
  • The triadic review stage's cost and latency impact are not individually quantified; the $0.14 average bundles all stages.
  • The paper text provided is truncated before the bug localization stage details and ablation results, so component-level quantification cannot be fully assessed from the supplied excerpt.
  • Docker-based sandboxing limits portability to environments where container execution is available.

Relevance to Harnesses / Meta-Harnesses

ReProAgent is a concrete instance of a tool-augmented multi-stage meta-harness: it does not execute a task directly but orchestrates a pipeline of specialized agents/stages, each equipped with distinct tools, over a shared environment (the repository + Docker sandbox). The sequential-thinking controller that governs branching, revision, and continuation across stages is structurally analogous to a meta-harness loop that routes subtask results into subsequent stages. The pattern of wrapping a domain-specific workflow (bug reproduction) inside an agent framework with reflection, retrieval, and runtime interaction toolsets is directly relevant to research on how harnesses should decompose long-horizon software engineering tasks, manage tool invocation safely, and couple static analysis with dynamic execution feedback.