arXiv preprint ·
Comparing the Pre-trained Models of Source Code by Re-pre-training under a Unified Setup
- Published
- Source
- arXiv preprint, posted before peer review
- Published in
- IEEE Transactions on Neural Networks and Learning Systems
- DOI
- 10.1109/tnnls.2023.3308595
- Original
- Read it at arXiv
- Cited by
- 6 works, as OpenAlex counts them
- Topics
Abstract
Recent years have seen the successful application of large pre-trained models of source code (CodePTMs) to code representation learning, which have taken the field of software engineering (SE) from task-specific solutions to task-agnostic generic models. By the remarkable results, CodePTMs are seen as a promising direction in both academia and industry. While a number of CodePTMs have been proposed, they are often not directly comparable because they differ in experimental setups such as pre-training dataset, model size, evaluation tasks and datasets, etc. In this paper, we first review the experimental setup used in previous work and propose a standardized setup to facilitate fair comparisons among CodePTMs to explore the impacts of their pre-training tasks. Then, under the standardized setup, we re-pre-train CodePTMs using the same model architecture, input modalities, and pretraining tasks, etc., as they declared and fine-tune each model on each evaluation SE task for evaluating. Finally, we present the experimental results and make a comprehensive discussion on the relative strength and weakness of different pre-training tasks w.r.t. each SE task. We hope our view can inspire and advance the future study of more powerful CodePTMs.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Software Engineering
Verifying Autoware Controller Node by Combining Model Verification and Runtime Monitoring
When Agent Evaluation Becomes Unreliable: An Empirical Study of Oracle Failures and Configuration-Dependent Measurement in an LLM Agent Benchmark
PassSum: Leveraging Paths of Abstract Syntax Trees and Self-Supervision for Code Summarization
SafeDSE: Joint Exploration of Safety Architecture Patterns, Mapping, and Scheduling for Dataflow Applications With Safety Arguments (Extended Version)
EvoWeb: A Benchmark for Coding Agents on Evolving Web Applications
Get the few papers worth your time in Software Engineering every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.