arXiv preprint ·
RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?
- Published
- Source
- arXiv preprint, posted before peer review
- Published in
- arXiv
- DOI
- 10.48550/arxiv.2609.28850
- Original
- Read it at arXiv
- Cited by
- 0 works, as OpenAlex counts them
- Topics
In brief
AI Agents vs. ML Paper Reproducibility
This paper introduces RECLAIM, a benchmark that tests AI agents on their ability to reproduce results from 100 NeurIPS 2025 papers. By categorizing releases into tiers based on their completeness, the authors provide a structured way to evaluate the capabilities of these agents.
What they found: The RECLAIM benchmark revealed that AI agents struggle with reproducing results from machine learning papers, especially as the completeness of releases decreases. Success rates were 41% for Run-tier, 27% for Retrain-tier, and a mere 15% for Reimplement-tier papers. On average, agents used only 29% of their GPU-hour budget before stopping, indicating inefficiencies in their reproduction attempts.
AI-written summary. Check anything you rely on against the original paper.
Abstract
Reproducing a machine learning paper involves most research steps, from installing software and debugging to running experiments, work that AI agents increasingly do. We introduce RECLAIM, a benchmark of 100 NeurIPS 2025 papers that can be rebuilt yearly from new conferences. For each paper we fix in advance the result to reproduce, what counts as a successful reproduction, and a GPU-hour budget. An agent must reproduce that result using the paper and whatever its authors released. What the authors released decides the difficulty tier. Run-tier releases include code, data, and weights; Retrain-tier releases lack weights, so the agent trains the model; Reimplement-tier releases lack code, so the agent writes it. A separate language model grades runs from logs and outputs rather than agents' reports. We run four agents once per paper; the best agent in each tier reproduces only 41% of Run-tier papers, 27% at Retrain, and 15% at Reimplement, where every agent does worst. Failed attempts use on average 29% of their budget, so most stop with budget left. The most common agent error is writing the method without checking any part against the paper's numbers, in 63 of 400 runs.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Artificial Intelligence
Verified Learning for Compiler Optimization: An LLM-Guided Architecture with Formal Control
pytest-gpu-proof: Enabling Cloud-CPU Continuous Integration for GPU Code with Local GPU Attestation
From PyTorch to the NPU: LLM-Agent-Driven Model Conversion Across Heterogeneous Inference Runtimes
Specification-Driven Benchmarking for Automated Program Repair From Static Corpora to Executable Specifications
Specifying and Maintaining Agentic Workflows: An Empirical Study of GitHub Agentic Workflows
Get the few papers worth your time in Artificial Intelligence every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.