arXiv preprint ·
OneCodeReviewBench: Typed Decisions for Code Review and Search
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Code review requires finding functional bugs, memory-safety errors, security weaknesses, and performance problems across languages and software stacks. Tools can express bounded parts of this work as decisions about supplied code. We present OneCodeReviewBench, 1,968 synthetic cases across 12 programming languages and YAML, with Boolean, categorical, and ordered answers. Explicit contracts specify runtime assumptions, source scope, and answer criteria; separate evidence records support reference review. One author reviewed 150 cases; two additional engineers reviewed 100 of those cases. We compare ten model--adapter systems on the same cases and contracts: OpenAI Decisions reaches 81.3% Boolean balanced accuracy; Kev-27B reaches 76.8% Choice accuracy and Score mean absolute error of 0.367. Excluding two families with incomplete contracts preserves these leaders. Additional comparisons cover 586 cases with raw symbol search and 40 with a bounded Python import resolver; the resolver matches all 40 references within its supported profile. A versioned artifact reproduces these analyses and retains the 85 cases with matching engineer answers; the reported review subset excludes two incomplete-contract cases. Remaining reviewer disagreements and repaired inputs require resolution; repository-scale transfer and matched latency/cost evaluation remain future work.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Software Engineering
PMU Foundation Model Bridge: A Tool for Connecting Synchrophasor Streams to Time Series Foundation Models
How Do Small Coding Agents Use Code Graphs and Embeddings?
Securing Agent-Generated Code with Its Own Development History
BoardSafety: AI-Ready Data from Manufacturer Documentation for Development-Board Safety
Bug Severity Classification with Fine-Tuned Open LLMs: A Paired Comparison with TF-IDF and CodeBERT Baselines on Eclipse Reports
Get the few papers worth your time in Software Engineering every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.