arXiv preprint ·
Finding the Pivots to Repair Broken Source-Test Evolution Using LLMs
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Repairing obsolete tests in response to source code evolution is critical for test suite reliability and software quality assurance. While large language models (LLMs) have emerged as promising tools for automating many software development tasks, the effective supply of context and the orchestration of repair pipelines in this source--test co-evolution scenario remain poorly understood. Moreover, although reasoning is found helpful for LLMs on some complex coding tasks, it remains unclear which reasoning steps actually contribute to effective test repairs and how the insights learned from LLMs' behaviors benefit coding agents. In this paper, we present a systematic empirical study of test repair for source-test co-evolution empowered by LLMs, investigating the effective setups and typical LLM behaviors. To support this study, we construct a new benchmark of 500 real-world source-test co-evolution cases from commits submitted between May 2024 and April 2026 across 15 highly-starred open-source Java projects. Our experimental results reveal several key insights for effective LLM-based test repair: the complementary benefits of combining test failure symptoms, source code changes, and supportive code context, as well as the effectiveness of self-repair rounds. We further derive a common reasoning action taxonomy from LLMs' reasoning traces, which proves instrumental in guiding LLMs to repair tests more effectively while at a lower financial cost. Applying our findings to the Codex agent further improves its repair success rate on challenging cases by 66.7%.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Software Engineering
APG: Agentic Static-Analysis Infrastructure on LLVM-IR Graph Databases
Validation-Gated ML Pipeline Synthesis with Expert-Grounded Self-Evolving Skills
Toward an Evidence-Based Template for Agent Instruction Files: An Empirical Study of Adoption, Consistency, and Maintenance
Are We Just Applying LLMs? What SE Research Studies, Underexplores, and Could Do Next in the LLM Era
Fault-Free Profiles and Decision-Level Training for Root Cause Analysis via Small Language Models
Get the few papers worth your time in Software Engineering every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.