arXiv preprint ·
Accountable and uncertainty-aware evaluation of sensor-based AI under distribution shift: devices, subjects, and nearly three years underground
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
In brief
Evaluating AI in Real-World Conditions
The authors propose a novel evaluation protocol that quantifies uncertainty and establishes clear reference levels for performance. This method is tested on geomagnetic localization using smartphones in underground mines, showcasing its effectiveness in real-world conditions.
What they found: The proposed evaluation protocol demonstrated a present-conditioned precision of 0.39, which is 16.5 times better than chance performance. The method effectively revealed performance variability across 42 location classes, with a margin of +/-0.08. Notably, the analysis showed that relying solely on mean performance can be misleading, as evidenced by a bimodal configuration that performed poorly at the 5% quantile despite passing mean tests.
AI-written summary. Check anything you rely on against the original paper.
Abstract
Sensor-based AI systems are rarely operated under the conditions under which they were trained: devices, personnel and recording epochs change, and each change degrades performance in ways a random train-test split cannot reveal. We propose a staged, accountable evaluation protocol that treats the evaluation of a deployed model as a measurement with declared reference levels and a quantified uncertainty. Four cumulative generalisation stages hold out devices, subjects and time. Each stage is judged on quantiles of repeated trainings against chance references with the correct class count, an out-of-present-scope rate exposes silent misdirection towards classes that are no longer present in deployment relative to training, and an explicit decision rule ties roll-out decisions not to means but to 5% quantiles. We demonstrate the protocol on infrastructure-free geomagnetic localisation with smartphone-based recurrent classifiers in two real underground mines, including a replication of the scheme's training stages at the second site. Unchanged models are re-evaluated on data recorded 34 months after the training campaign, on a device generation unknown at training time and with a held-out surveyor. The 5% quantile of their present-conditioned precision there is 0.39 over 299 repeated trainings, 16.5 times the chance level; across the composition of the 42 reachable location classes the figure varies by +/-0.08, several times the spread between repeated runs. Repeated trainings of a single configuration show why means mislead: a bimodal configuration passes a mean-based test decisively while its 5% quantile lies more than an order of magnitude below chance.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Machine Learning
Distribution-free inference on the number of changepoints
Optimal Value Inference for Reinforcement Learning
Geographically Regularized AUC-Maximizing Personalized Federated Learning
Dynamical Non-compensatory Multidimensional IRT Model Using Variational Approximation
Instability in Patient Clustering: A Multiverse Analysis of Unsupervised Clustering in the CENTER-TBI cohort
Get the few papers worth your time in Machine Learning every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.