arXiv preprint ·
Noisy Issues, Noisy Results: Mislabeling in Information Retrieval-based Bug Localization
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Information Retrieval-based Bug Localization (IRBL) leverages bug reports to identify faulty source files. Evaluations of IRBL approaches primarily rely on labeled issues from issue tracking systems. However, prior research has shown that issue labels are frequently incorrect in open-source projects, threatening the validity of IRBL evaluations. Existing studies have primarily examined traditional IRBL techniques on small datasets, overlooking modern approaches and the partial overlap between noisy and cleaned datasets. In this paper, we conduct a large-scale empirical study to investigate the prevalence and impact of mislabeled issues in IRBL benchmarks. We analyze five benchmark datasets comprising over 55,000 issue reports and find that, on average, 34.82% of issues are mislabeled. We further evaluate six IRBL techniques, spanning traditional, deep learning-based, and large language model-based approaches using the widely adopted Ye et al. benchmark and statistical methods that account for dataset overlap. Our results show that removing mislabeled issues from evaluation datasets significantly reduces measured performance across all six techniques, with Mean Reciprocal Rank decreasing by 4.68%-21.74%. Although mislabeling does not change the relative ranking of techniques, it affects the magnitude of their performance differences. These findings highlight the importance of reliable evaluation data for ensuring valid and meaningful comparisons of IRBL techniques.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Software Engineering
Validation-Gated ML Pipeline Synthesis with Expert-Grounded Self-Evolving Skills
DARE: Difficulty Analysis for Repair Enhancement
Are We Just Applying LLMs? What SE Research Studies, Underexplores, and Could Do Next in the LLM Era
A Hybrid Approach for Automated Security Pattern Recovery
Planning-Coding Separation in LLM-Based Code Generation: An Empirical Study
Get the few papers worth your time in Software Engineering every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.