arXiv preprint ·
Data Quality in RVL-CDIP: A Systematic Audit
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
RVL-CDIP is the de facto standard benchmark for document classification, and reported accuracy on it is now 98.8%. In this paper, we audit RVL-CDIP for data quality issues and find that approximately 10.5% of its 400,000 documents have erroneous labels and that 35.2% of the test set has a duplicate or near-duplicate in the training set. Identification codes on the pages are sufficient to classify at 69% accuracy, and a state-of-the-art model loses 11 points without them. Image-only models trained on RVL-CDIP lose 16 to 28 points on documents from other sources, and the choice of OCR engine changes accuracy by 4.1 to 8.3 points. We find that nearly 16,000 RVL-CDIP documents contain sensitive personal data, and produce de-identified versions of these documents. We re-benchmark 25 models, and with all defects remediated, the 13 that we retrain are 0.9 to 2.9 points more accurate than in the original condition; the most accurate of them, LayoutLMv3, is 96.5% accurate. We release the corrected corpus, the annotations and the out-of-distribution data.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Computer Vision
CRM-Agent-Bench: Benchmarking Tool-Using Agents in Stateful Enterprise Environments
Examining form-based linguistic productivity in people and large language models
X-Dubbing: Speech-to-Speech Translation under Temporal and Visual Constraints
Perturbation-Based Data Selection for Medical Question Answering
How Much Do You Trust A Dog? Quantifying Hint-Following in Reasoning Language Models
Get the few papers worth your time in Computer Vision every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.