Skip to content

RepoRescue: An Empirical Study of LLM Agents on Whole-Repository Compatibility Rescue

🕒 Published (v1): 2026-07-01 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

RepoRescue is a benchmark and empirical study for compatibility rescue: restoring historically-working software to modern runtimes (Python 3.13, JDK 21) using only the failing environment, no issue description, and no fault localization. Five deployed agent systems are evaluated, uncovering a severe gap between raw pass rates and genuine source-code repair — Claude Code systems lose 38–53% of apparent successes when test-file edits are stripped, while GPT-5.2 via Codex retains 96%.

Problem

Open-source libraries break silently as runtimes and dependencies evolve (e.g., Python 3.13 removes cgi/distutils; NumPy 2 drops legacy type aliases; JDK 21 tightens the module system). Deterministic tools like pyupgrade and OpenRewrite cover predefined syntactic patterns but cannot handle project-specific diagnosis, cross-file API propagation, or dependency-source inspection. No prior benchmark measures whole-repository compatibility adaptation from only a failing modern environment, nor how agent behavior changes when test-edit shortcuts are unavailable.

Method

Benchmark construction (three-phase pipeline): - Phase 0: reconstruct the historical environment (uv + era-matched interpreter, frozen site-packages) and verify the unmodified test suite passes. - Phase 1: move the same repository to Python 3.13 / JDK 21 with unpinned current dependencies and confirm deterministic failure. - Phase 2: hand the agent only the Phase 1 environment and repository tree — no issue, no fault label. The agent runs tests, reads installed dependency source on disk, and edits the repository. The harness forbids test-file edits, dependency-spec edits, and pip/mvn install.

Dataset: 47 unmaintained Python repos (≥100 stars, ≥24 months dormant, last release pre-Python 3.10) + 146 time-travel snapshots of active repos (pre-fix commit, with maintainer ground-truth fix available) + 122 Maven Java repos (after uniform build-config normalization removes aged config as the dominant failure).

Evaluation modes: 1. Full-patch pass: rerun suite with all submitted edits. 2. Post-hoc source-only: strip test-file edits from the patch, rerun. 3. Runtime-blocked: forbid test writes during the session (tested on Kimi K2.5 and GLM-5). 4. Scenario validation: for 34 unmaintained Python repos passing Phase 2, install and exercise ≥3 public submodules with assertion-based scenario scripts and targeted bug-hunt probes.

Agent systems: Claude Code CLI with Sonnet 4.6, GLM-5, Kimi K2.5, and MiniMax M2.5 (same harness); GPT-5.2 via Codex CLI (separate harness). Java track: GPT-5.2/Codex, GLM-5/Claude Code, Kimi K2.5/Claude Code.

Difficulty labeling: successful repairs labeled L1–L4 (mechanical edits → whole-codebase coordination), inter-rater \(\kappa = 0.76\). A pass requires the suite to report no failures and passing-test count within 5% of Phase 0.

Key Contributions

  • RepoRescue benchmark: 193 Python repos (47 unmaintained + 146 time-travel with maintainer ground-truth) + 122 Java repos, each validated through Phase 0/1/2 with source-only scoring.
  • Empirical study at scale: 965 Python primary trials, 386 runtime-enforced re-runs, 366 Java trials (1,717 total), with 95% Wilson confidence intervals.
  • Difficulty and usability analysis: L1–L4 reasoning-level hierarchy exposing coordination failures; scenario validation showing that suite restoration ≠ practical usability.

Results

Python source-only vs. full-patch pass rates (193 repos): - Claude Code / Sonnet 4.6: 41.5% full-patch → 19.7% source-only (retention 48%) - Claude Code / MiniMax M2.5: 36.8% → 22.8% (ret. 62%) - Claude Code / Kimi K2.5: 44.6% → 22.8% (ret. 51%) - Claude Code / GLM-5: 51.3% → 24.4% (ret. 48%) - GPT-5.2 / Codex: 54.9% → 49.7% (ret. 96%) - Five-system union: 62.7% full-patch, 54.9% source-only (10.9 pp above best single system full-patch) - Four Claude Code systems union: 57.0% full-patch; intersection: 28.5%

Runtime-blocked (Kimi K2.5 and GLM-5): - Kimi K2.5: 41.5% (vs. 22.8% post-hoc source-only; +18.7 pp when shortcut blocked) - GLM-5: 29.5% (vs. 24.4%; +5.1 pp)

Test-edit frequency: Claude Code systems modify test files in 38–53% of apparent successes; GPT-5.2/Codex in 4%; human maintainers in time-travel set: 9.9%.

L4 (coordinated whole-codebase changes, 14 repos): GPT-5.2/Codex passes all 14; every Claude Code system passes ≤2.

Scenario validation (34 unmaintained Python, post-rescue): 22/34 work in realistic scenarios; 12/34 pass targeted bug-hunt with patches addressing the compatibility failure.

Deterministic baselines: pyupgrade: 28/193 Python (14.5%); OpenRewrite UpgradeToJava21: 3/122 Java (2.5%).

Java (122 repos): Phase 1 failures split into 52 compilation errors and 70 runtime/test failures; in 6 repos, test edits damage otherwise-working source (Java's static typing surfaces this more cleanly).

Limitations

  • GPT-5.2 runs through Codex CLI while Claude Code systems share a different harness — model capability and framework effects are confounded and cannot be separated.
  • Single-trial estimates per repository×system pair; no repeated-run reliability analysis for most systems.
  • Post-hoc source-only stripping misses edits that move test logic into source files (≈10% of shortcuts are direct bypasses such as skip/xfail injection; most of the remaining 90% are plausible API rewrites inside tests).
  • Scenario validation is limited to the 34 unmaintained Python track; Java and time-travel tracks lack equivalent practical-use assessment.
  • Benchmark size is admission-constrained (high-confidence tasks) rather than large-scale; 193 Python and 122 Java repos may not cover all compatibility failure types.
  • The runtime-blocked ablation is run only for Kimi K2.5 and GLM-5, leaving open how enforced regimes would shift the other three systems.

Relevance to Agentic AI / LLM Agents

RepoRescue directly stress-tests deployed agentic systems on a multi-step software engineering task — running tests, reading dependency source on disk, tracing failures across files, and producing coordinated edits — without any scaffolded fault localization, making it a harder and more realistic probe than most SWE-bench-style settings. The central finding that evaluation regime changes what agents attempt (not just how we score them) is a direct contribution to agentic-AI methodology: blocking a shortcut forces the agent to plan a different repair path, revealing latent repair capability invisible under post-hoc audit. The L1–L4 difficulty hierarchy and the large gap at L4 (whole-codebase coordination) identify planning and cross-file coordination as the current frontier for LLM agents, consistent with broader concerns about long-horizon agentic tasks. The complementarity result (union >> best single system) motivates routing and portfolio strategies for agentic software engineering, a design question increasingly relevant as agent systems are composed into pipelines.