arXiv preprint ·
Large Language Model Turnover Undermines Screening for Artificial Intelligence-Assisted Scientific Writing
- Published
- Source
- arXiv preprint, posted before peer review
- Published in
- arXiv
- DOI
- 10.48550/arxiv.2610.11599
- Original
- Read it at arXiv
- Cited by
- 0 works, as OpenAlex counts them
- Topics
In brief
When AI-Writing Detectors Fall Behind
The authors tested detectors against 23 LLM versions from three vendors, using 4,000 pre-ChatGPT scientific abstracts and rewrites of those abstracts. They found that detection can collapse across model-generation boundaries, and that screening all versions involves a difficult trade-off between falsely flagging human writing and missing AI rewrites.
What they found: At a 1% false-flag rate on human abstracts, a detector trained on a vendor’s earlier versions caught above 99% of rewrites just before the sharpest generation boundary, but only 3.8% just after it. Across screening scenarios covering all 23 versions, one approach flagged one in eight human-written abstracts, while another missed one in three rewrites from the newest version.
AI-written summary. Check anything you rely on against the original paper.
Abstract
Journals and conferences have begun to screen submitted manuscripts for text written using large language models (LLMs). The reliability of this screening rests on benchmark evaluations against a fixed set of LLM versions, while the versions in actual use keep changing. Here we quantify how this LLM turnover affects the screening of scientific manuscripts. We paired 4,000 pre-ChatGPT abstracts from the Proceedings of the National Academy of Sciences with their rewrites by 23 LLM versions from three vendors, released between June 2023 and August 2026. We then trained detectors under maintenance scenarios ranging from a detector retrained on every new version to one trained once and never updated. Detectors trained only on a vendor's past versions can collapse at the boundaries between model generations: calibrated to falsely flag 1% of human-written abstracts, they catch above 99% of rewrites just before the sharpest boundary and 3.8% just after it. Detectors trained on later versions can also miss rewrites of earlier ones. Vocabulary differences between versions largely track where detection transfers and where it fails. In the two screening scenarios we simulated, screens covering all 23 versions either flagged one in eight human-written abstracts or missed one in three rewrites of the newest version. Indeed, a commercial detector missed most rewrites of the version just after the sharpest boundary while flagging almost no human-written abstracts. Research-integrity policy should therefore treat the benchmark accuracy of a detector as provisional, to be re-verified with every LLM release, including earlier versions.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Computation & Language / NLP
SAIL: Scientific Agentic Intelligence via a Science-Aware Loop
Do you "see" what I "see"? A Multi-panel Visual Question Answer Dataset for Large Language Model Chart Analysis
A Stateful Research Information Infrastructure for Scientific Discovery
Wieszcz-XIX: A 3.1-Billion-Word Corpus of Pre-1918 Polish and Temporally Bounded Language Models Trained From Scratch
Gugak-VocalSet: A Multi-Genre Studio-Quality Dataset of Korean Traditional Singing with Sigimsae Annotations
Get the few papers worth your time in Computation & Language / NLP every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.