Skip to content

SWE-Review: Closing the Loop on Issue Resolution with Agentic Code Review

🕒 Published (v1): 2026-07-07 00:00 UTC · Source: HuggingFace · link

Why this paper was selected

SWE-bench lineage; closes open-loop PR generation with agentic review-revise cycle

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SWE-Review introduces an agentic code review framework that closes the open-loop gap in AI-driven pull-request generation: a reviewer agent explores a repository, emits a binary accept/request-changes decision, and produces structured diagnoses that guide iterative revision. Experiments on SWE-bench Verified show that this generate-review-revise loop nearly doubles resolve rates for weaker PR generators and enables effective test-time scaling.

Problem

AI coding agents produce PRs in one shot with no reliable mechanism to determine whether the issue was actually resolved or to diagnose how a failing patch should be fixed. Existing automated review systems are single-turn, fixed-context, and evaluated with text-overlap metrics that do not measure decision correctness or downstream revision utility—leaving a systematic benchmark, training data, and end-to-end evaluation gap.

Method

Given a repository checkout, a natural-language issue, and a candidate PR diff, a reviewer agent is deployed via the OpenHands-SDK scaffold. It actively gathers evidence—browsing files, tracing call chains, executing lightweight checks—before committing to a structured review report with two fields: (1) a binary accept/request-changes decision and (2) a diagnosis citing defect locations and actionable fixes. This report feeds a generate-review-revise loop: rejected PRs are returned to the original PR generator with the diagnosis for one revision round; approved PRs are accepted as resolved.

Three evaluation metrics are defined operationally against executable test suites: - Completion Rate (CR): fraction of parseable review reports produced. - Decision Accuracy (DA): binary decision vs. true resolve status; incomplete reviews scored at 0.5 (chance). - Resolve Rate after Revision (RRR): final resolve rate after one review-guided revision.

SWE-Review-Bench comprises 1,384 candidate PRs from 500 SWE-bench Verified issues, generated by three models (GLM-5 at 72.2% baseline resolve, Qwen3-Coder-30B-A3B at 50.9%, Qwen3-30B-A3B at 27.5%).

SWE-Review-Traj is a distillation dataset of 8,914 decision-correct agentic review trajectories sourced from SWE-rebench, produced by GLM-5 (thinking mode) as teacher, filtered by executable patch outcome, and validated semantically (Claude Opus 4.6 / GPT-5.4 judges, Cohen's \(\kappa = 0.72\)) and functionally (RRR under four feedback conditions).

Key Contributions

  • SWE-Review framework: formalizes repository-grounded agentic code review as the pivot of a generate-review-revise loop with operationally-defined metrics (CR, DA, RRR).
  • SWE-Review-Bench: 1,384-PR benchmark stratified across three PR-quality tiers, supporting controlled evaluation of reviewer decision accuracy and revision utility.
  • SWE-Review-Traj: 8,914 decision-correct agentic review trajectories for open reviewer training and research.
  • SWE-Review-8B / 30B: distilled open reviewers trained via SFT on SWE-Review-Traj, showing substantial CR and DA gains over base models.
  • Mixed training result: combining issue-resolution and review trajectories in a single SFT run improves direct resolve rate and produces a unified generate-review-revise agent.
  • Test-time scaling: trained reviewers outperform dedicated scalar verifiers for iterative review-guided revision.

Results

  • Iterative loop (Claude Opus 4.6 reviewer): resolve rate raised from 27.5% → 56.9% (+29.4 pp) for Qwen3-30B-A3B; 50.9% → 68.8% (+17.9 pp) for Qwen3-Coder-30B-A3B; 72.2% → 75.4% (+3.2 pp) for GLM-5.
  • Agentic vs. single-turn review (DA): agentic review scores 89.4% DA (Qwen3-30B-A3B split) vs. 73.8% for diff+context single-turn; gains widen monotonically from easy to hard patches.
  • Agentic vs. single-turn review (RRR): 52.6% vs. 44.1% (diff+context) for Qwen3-30B-A3B split.
  • SFT distillation (Qwen3-8B → SWE-Review-8B): CR rises from ~4% to 71–84%; DA improves 18–21 pp; RRR rises from 27.5% → 35.1% on Qwen3-30B-A3B split.
  • SWE-Review-30B-A3B: RRR 35.8% (+8.3 pp) on the hardest split.
  • Mixed training (issue-resolution + review, 2k each): RR improves 31.2% → 36.8% (+5.6 pp); RRR 31.2% → 41.8% (+10.6 pp) for Qwen3-8B.
  • Functional validation of diagnosis: no review → 3% RRR; decision-only → 8%; teacher diagnosis → 21%; oracle → 32%.
  • Test-time scaling: resolve rate raised from 22.9% → 38.4% over 4 revision rounds with trained reviewers.

Limitations

  • Reviewer model used in main experiments is Claude Opus 4.6 (closed, expensive); open distilled models lag substantially, especially on high-quality PR generators.
  • SWE-Review-Traj trajectories are generated by a single teacher (GLM-5), potentially biasing the training distribution.
  • RRR allows only one revision round per review cycle in the benchmark protocol; the multi-round loop results use a separate pipeline.
  • Diagnosis quality validation shows mean scores of ~3/5, indicating imperfect diagnoses even after filtering.
  • Stricter semantic filtering (score ≥ 3 from both judges) did not improve trained reviewer performance, suggesting diagnosis quality signal is noisy.
  • The benchmark is restricted to 500 SWE-bench Verified instances; generalization to other issue distributions is not directly evaluated.

Relevance to Agentic AI / LLM Agents

SWE-Review directly advances closed-loop agentic software engineering by replacing one-shot PR generation with an iterative generate-review-revise loop, a paradigm shift relevant to any agent system that produces artifacts requiring correctness verification. The finding that active, adaptive evidence gathering (tracing call chains, running lightweight checks) substantially outperforms fixed-context retrieval operationalizes a key principle for tool-using agents: the correct context cannot always be pre-specified and must be discovered during execution. The mixed-training result—that review trajectories transfer to improve issue-resolution capability—suggests that repository reasoning is a shared substrate across agentic coding roles, with implications for unified agent training. The test-time scaling via iterative reviewer-guided revision is a practically deployable alternative to scalar verifiers for best-of-N selection in coding agent deployments.