arXiv preprint ·
Rethinking SZZ Search Strategies: Lessons from Claude Code
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Identifying bug-inducing commits (BICs) is important for studying software bugs and maintaining software, yet the accuracy of existing SZZ methods remains limited. Although various improvements have been proposed, many SZZ methods still rely on code tracing, which searches for BICs by tracing code changed in a bug-fixing commit (BFC) back to earlier commits. However, this approach may miss the actual BIC when the code responsible for introducing a bug differs from the code modified to fix it. To overcome this limitation, we investigate whether BIC identification can be improved by exploring search strategies that are not constrained to start with code tracing. We use Claude Code, a coding agent that has demonstrated high BIC identification accuracy in prior research, to explore such strategies. Rather than prescribing a search process or providing predefined candidate commits, we allow Claude Code to independently investigate a Git repository. We evaluate its identification accuracy and search behavior on 79 Java cases from 70 repositories. In 79.7% of runs, Claude Code reviewed commit histories before inspecting individual code diffs. It used commit messages to select changes for closer inspection. With this approach, Claude Code achieves an F1 score of 0.771, significantly outperforming all ten evaluated SZZ baselines. Our failure analysis also reveals incorrect oracle labels and ambiguities in BIC definitions in the studied evaluation dataset. After reconstructing the evaluation dataset to address these issues, Claude Code achieves an F1 score of 0.938. A key lesson from Claude Code for improving SZZ search strategies is to use semantic information to find relevant commits and then examine their code changes, rather than starting with code tracing.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Software Engineering
HoldSpec: A Machine-Checked Model and Conformance Suite for the Payment Authorization Hold Lifecycle
Verifying Autoware Controller Node by Combining Model Verification and Runtime Monitoring
Specification-Based Testing of Large Language Models
PassSum: Leveraging Paths of Abstract Syntax Trees and Self-Supervision for Code Summarization
Taylor-Approximation-Based Repair of High Floating-Point Errors
Get the few papers worth your time in Software Engineering every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.