arXiv preprint ·
Evaluating the Performance of LLMs in Drafting NIH Data Management Plans
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Sharing research data is essential to promote transparency, enable reproducibility, and fuel scientific discoveries. To ensure consistent sharing, the National Institutes of Health (NIH) and other funding agencies require applicants to submit data management plans (DMPs) with their proposals. They aim to encourage researchers to consider their data management and sharing strategies early on. However, researchers often face challenges when developing DMPs due to a lack of knowledge about data-sharing practices. Here, we evaluated the performance of Large Language Models (LLMs) in drafting DMPs compliant with the 2023 NIH guidelines. The goal was to investigate whether LLM-supported DMP creation could reduce the burden on researchers and librarians. We compared Llama and GPT performance against NIH-provided DMP samples. We also asked domain experts to evaluate LLM-generated DMPs. Overall, we found that LLMs can accelerate the creation of initial drafts but still require expert review before submission to ensure clarity and completeness. Moreover, open-source LLMs may not be as ready off-the-shelf as commercial ones for drafting funder-compliant DMPs.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Digital Libraries
Terms of Care: Designing for Participation in Data Governance by Disability Communities
Latin American Studies since the 1980s: From Apocalyptic Dreams to Assimilationist Consolidation
Auditing AI peer reviewers: a benchmark on real scientific papers
Building and Auditing Chemical Datasets with LLMs: A General LLM-as-a-Judge Framework Applied to Homogeneous Catalytic PET Depolymerisation
How did data sharing evolve in the decade since the FAIR Principles?
Get the few papers worth your time in Digital Libraries every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.