arXiv preprint ·
PremiseCheck: Efficient Fact-Checking of Human and AI Peer Reviews
- Published
- Source
- arXiv preprint, posted before peer review
- Published in
- arXiv
- DOI
- 10.48550/arxiv.2610.21046
- Original
- Read it at arXiv
- Cited by
- 0 works, as OpenAlex counts them
- Topics
In brief
Fact-Checking Reviews for Less
The paper introduces ReviewFact, a benchmark for factual errors in review comments across NLP, machine learning, and the natural sciences, with cost and latency measured alongside accuracy. It also proposes PremiseCheck, which turns each comment into explicit claims before checking them against the paper, and reports accuracy comparable to GPT-5.5 at substantially lower cost.
What they found: On the ReviewFact benchmark, PremiseCheck achieved accuracy comparable to GPT-5.5 when checking comments individually across NLP, machine learning, and the natural sciences. It did so at about half the cost across the three fields. For NLP reviews specifically, the abstract reports about one-quarter of the cost and less than half the latency.
AI-written summary. Check anything you rely on against the original paper.
Abstract
Peer review is under strain from the growth of submissions, and review comments written by humans or AI can misstate what a paper says. We introduce ReviewFact, a benchmark that labels such factual errors in review comments under one definition across NLP, machine learning, and the natural sciences, and that measures cost and latency alongside accuracy. A frontier LLM checks comments well but must read the whole paper for each check, while cheap decision models, which score whether a statement holds in a text, fall far short on whole comments that mix several implicit assertions. PremiseCheck is a framework that combines the two. An LLM rewrites each comment as explicit premises without reading the paper, and a decision model checks each premise against the paper. When each comment is checked on its own, PremiseCheck reaches accuracy comparable to GPT-5.5 across all three fields at about half its cost, and on NLP reviews at a quarter of its cost and less than half its latency.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Computation & Language / NLP
Learning When to Think: Utility-Guided Adaptive Reasoning for Efficient Text Retrieval
Self-Evolving Coding Agents via Harness-Policy Optimization
INCONVENIENT: A Benchmark to Assess LLM Generalization Capabilities from Convenience Samples
Prune the Leaky Neurons: Mitigating Private Data Leakage in Fine-Tuned LLMs via Joint-Embedding-Guided Pruning
ReviewPilot: A Multi-Agent System for Screening and Data Extraction in Systematic Reviews
Get the few papers worth your time in Computation & Language / NLP every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.