RepoRescue: An Empirical Study of LLM Agents on Whole-Repository Compatibility Rescue

Zhihao Lin ,

Mingyi Zhou ,

Zhensu Sun ,

Yizhuo Yang ,

Renyu Yang ,

David Lo ,

Li Li

Agent Tool Learning Agent Eval Benchmarks AI Systems and Hardware CACRR PVDAR

2026年07月01日

开源库与工具被广泛复用，但维持其兼容性成本高昂。一旦原维护者退出，那些原本有价值的代码仓库便可能因运行时环境和依赖库的持续演进而逐渐失效。我们研究了大语言模型（LLM）智能体是否能够将老旧仓库适配至现代开发环境——我们将这一任务称为“兼容性救援”（compatibility rescue）。与缺陷修复不同，兼容性救援的起点是一个在其原始环境中曾正常运行、却在生态系统发生演化（ecosystem drift）后出现故障的代码仓库。RepoRescue 仅向智能体提供待救援的仓库及其当前报错的现代运行环境；智能体必须自主诊断失败原因、定位受影响的代码位置，并生成一份源代码层面的修复补丁，以使该仓库的历史测试套件重新全部通过。我们构建了 RepoRescue 基准数据集，包含 193 个 Python 仓库和 122 个 Java 仓库，每个仓库均经人工验证：在其历史环境中全部测试通过，而在现代化升级后全部测试失败。我们在 Python 上评估了五种已部署的智能体系统，在 Java 上评估了三种。除整体补丁通过率（full-patch pass rate）外，我们还进一步开展三项精细化评估：（1）移除补丁中对测试文件的修改后重新运行，以衡量纯源码级（source-only）修复能力；（2）引入运行时强制约束机制（runtime-enforced regime），彻底禁止对测试文件的任何编辑；（3）针对经救援后测试套件已全部通过的仓库，开展实际可用性验证（practical validation）。结果表明：Claude Code 系列系统即便在明确提示“不得修改测试文件”的情况下，仍会不时擅自编辑失败的测试用例；而在启用运行时强制阻断机制后，Kimi 系统仍能成功完成 41.5% 的仓库救援任务。各系统能力具有互补性：若取其修复结果的并集，整体成功率可达 62.7%，较表现最优的单个系统高出 10.9 个百分点。问题难度高度集中于跨文件协同修改——在 14 个需协调整个代码库多处文件共同变更的仓库上，GPT-5.2 至 Codex 系列模型全部实现 100% 成功修复（即全部 14 个），而所有 Claude Code 系统最多仅能修复其中 2 个。最后需强调：测试套件全部通过仅是初步信号。在 34 个经救援后测试套件已全部通过、且长期无人维护的 Python 仓库候选对象中，有 22 个在真实场景下可正常运行；另有 12 个通过了“缺陷排查”（bug-hunt）验证——其补丁确实精准定位并解决了兼容性故障的根本原因。RepoRescue 基准全面覆盖四大维度：源码级审计（source-only auditing）、运行时强制约束（runtime enforcement）、实际可用性验证（practical validation）以及可解释的推理过程标注（reasoning labels）。

Open-source libraries and tools are widely reused, but compatibility maintenance is expensive. Once maintainers leave, useful repositories can stop working as runtimes and dependencies evolve. We study whether LLM agents can adapt old repositories to modern environments, a task we call compatibility rescue. Unlike bug repair, compatibility rescue starts from a repository that worked in its original environment but fails after ecosystem drift. RepoRescue gives agents only the repository and its failing modern environment; the agent must diagnose the failure, locate affected code, and produce a source-code rescue that restores the historical test suite. We build RepoRescue from 193 Python and 122 Java repositories, each verified to pass historically and fail after modernization. We evaluate five deployed agent systems on Python and three on Java. Beyond full-patch pass rate, we rerun patches after removing test-file edits to measure source-only repair, add a runtime-enforced regime that blocks test edits, and validate practical use for repositories whose suites pass after rescue. We find that Claude Code systems sometimes edit failing tests even when prompted not to; with runtime blocking, Kimi still rescues 41.5% of repositories. Systems are complementary: their union reaches 62.7%, exceeding the best single system by 10.9 points. Difficulty concentrates in cross-file coordination: on 14 repositories requiring coordinated whole-codebase changes, GPT-5.2 through Codex passes all 14, while every Claude Code system passes at most two. Finally, a passing suite is only an initial signal: among 34 unmaintained Python candidates whose suites pass after rescue, 22 work in realistic scenarios and 12 pass bug-hunt with patches that address the compatibility failure. RepoRescue benchmarks compatibility rescue with source-only auditing, runtime enforcement, practical validation, and reasoning labels.

许愿