arXiv preprint ·
Performance metrics in machine learning: a scenario-based guide from the confusion matrix to calibration, regression and uncertainty
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
In brief
When a High Score Misleads
This tutorial turns those pitfalls into 15 controlled scenarios, varying one factor at a time across classification, calibration, regression, uncertainty, and consistency checks. It also organizes 22 confusion-matrix metrics in a prevalence-coded map and provides a goal-based metric table and reporting checklist.
What they found: The scenarios show why accuracy can hide poor performance on rare positives: at prevalence 0.010, an always-negative rule achieved 0.990 accuracy but only 0.500 balanced accuracy. Holding sensitivity and specificity fixed, precision fell from 0.889 to 0.008 as prevalence dropped from 0.5 to 0.001.
AI-written summary. Check anything you rely on against the original paper.
Abstract
Performance metrics can yield different conclusions from the same predictions. This methodological tutorial explains the effects of class prevalence, decision thresholds, aggregation rules and software conventions through controlled scenarios. Twenty-two confusion-matrix metrics are organized in a prevalence-coded map. Fifteen scenarios cover binary and multi-class classification, score discrimination, probability calibration, regression, uncertainty and consistency checks. Inputs are fixed in released files, and one factor is varied in each scenario. All 205 verification checks passed; 202 were independent and 3 were internal. An always-negative rule achieved an accuracy of 0.990 at a prevalence of 0.010, with balanced accuracy of 0.500. At fixed sensitivity and specificity, precision fell from 0.889 to 0.008 as prevalence fell from 0.5 to 0.001. Three probability variants shared a ROC-AUC of 0.825 but had calibration slopes of 2.08, 1.04 and 0.52. With 10 participants, participant-majority accuracy of 0.700 had a 95% interval of 0.348-0.933. Fold-averaged scores were falsely flagged as inconsistent under an inappropriate pooled-test-set model. A goal-based metric table and a reporting checklist are provided. The scenarios illustrate mechanisms and do not estimate their frequency in published studies. A literature audit and a real-data demonstration are specified as future protocols and were not executed.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Machine Learning
Using Transformers for Teaching and Learning via Symbolic Distillation of Neural A-life Agents
TriAttention: Unified Sparse Mixed-Precision Attention for Long-Context LLM Inference
Evolutionary Adaptive Neighborhood Learning: A Genetic Algorithm for Instance-Dependent k Optimization in Nearest Neighbor Classification
Reweighting is not Harmless: Understanding the Role of Dataset Characteristics
Energy Measurement and Optimization of TinyML in Resource-Constrained Embedded Systems: A Systematic Mapping Study
Get the few papers worth your time in Machine Learning every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.