arXiv preprint ·
One Strategy Does Not Fit All: Rethinking Data Labeling for Just-in-Time Vulnerability Prediction
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Just-In-Time Vulnerability Prediction (JIT-VP) identifies security risks in code changes at commit time, enabling early feedback before vulnerabilities enter software systems. However, its practical reliability is challenged by the difficulty of constructing accurate labels. Multiple strategies with different trade-offs have been proposed to identify vulnerability-fixing commits (VFCs) and vulnerability-introducing commits (VICs) for dataset construction, leading to substantially different datasets. This makes it difficult to separate model performance from dataset construction. Moreover, real-world quality assurance requires not only classification but also prioritizing high-risk commits and localizing vulnerable causes. This paper presents an extensive empirical study of how labeling strategies affect these downstream tasks. We constructed 28 datasets using four VFC and four VIC labeling strategies across OpenSSL and Linux kernel, then evaluated six representative JIT-VP models for classification, prioritization, and localization. Our results show that labeling effectiveness is strongly task-dependent. Manual VFC labeling combined with vulnerability-specific VIC labeling provides the most reliable performance for classification and prioritization, significantly outperforming 11 of 15 alternatives. Meanwhile, precision-first VFC labeling combined with LLM-assisted VIC labeling performs best for localization, significantly outperforming 7 of 15 alternatives. These findings demonstrate that no single labeling strategy is universally optimal and highlight the importance of task-oriented dataset construction.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Software Engineering
Validation-Gated ML Pipeline Synthesis with Expert-Grounded Self-Evolving Skills
DARE: Difficulty Analysis for Repair Enhancement
Are We Just Applying LLMs? What SE Research Studies, Underexplores, and Could Do Next in the LLM Era
A Hybrid Approach for Automated Security Pattern Recovery
Planning-Coding Separation in LLM-Based Code Generation: An Empirical Study
Get the few papers worth your time in Software Engineering every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.