arXiv preprint ·
Neyman-Pearson Classification with Indecisions: Simultaneous Control of Selective Type I and Type II Errors
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Binary classification often requires controlling both type I and type II error rates at prespecified levels. When the class distributions overlap substantially, the two error limits may be incompatible if every observation must be classified. We study Neyman-Pearson classification with indecisions (NPI), which permits uncertain observations to remain unclassified and assesses selective type I and type II errors among definitive classifications within each class. For an independently trained score, we develop a model-free calibration procedure that combines all candidate threshold pairs passing calibration into a rule whose decision region contains that of every passing candidate. We couple the entire threshold search to a common martingale within each class and compute its finite-horizon crossing probability exactly, avoiding a multiplicity adjustment for the number of candidate pairs. The resulting classifier satisfies both population selective error limits simultaneously with user-specified high probability in finite samples, without assumptions on the accuracy of the fitted score. We bound the additional indecision due to calibration and evaluate the procedure in simulations and applications to recidivism and credit-default prediction.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Methodology
Same Predictions, Different Harms: Causal Auditing of Patient World Models
Calibration Assessment for Nested Uncertainty Sets
Causal Lag Structure Discovery in Confounded Time Series via Orthogonalized Adaptive Estimation
Jumping up and down: Denoiser diffusion models for discrete ordinal data
Errors of LLM-Assisted Literature Retrieval in Environmental Science: A Comparison Study of Abstract versus Full-text Based Prompts
Get the few papers worth your time in Methodology every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.