arXiv preprint ·
Benchmarking uncertainty quantification for engineering regression under graded distribution shift
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Machine learning models in engineering are often used beyond their training conditions, where reliable uncertainty quantification (UQ) matters most. Existing comparisons of regression UQ methods rely on random splits, which assess calibration only in distribution. We introduce a three-rung shift ladder (in distribution, soft shift and hard shift) built along physically meaningful shift variables, and evaluate 12 methods from five families on seven public engineering datasets: five extrapolation settings, an unseen turbofan fault mode and a winter air-quality sensor deployment. All tuning uses source data only. In distribution, conformal methods cover at least 0.946 of test points on every dataset. Under hard shift, mean coverage falls to 0.40-0.75 across the 11 methods that always return an interval, and no systematic difference in method ranks is detected (Friedman p = 0.63). Weighted conformalised quantile regression stays near nominal on the points it answers, but abstains on 85\% of hard-shift points on average and covers only 0.67 of the few Concrete points it answers. Source-validation gains predict improvement in distribution and under soft shift, but not under hard shift (Spearman's rho = 0.04). Engineering evaluations should split calibration data by deployment unit, report subgroup coverage and sharpness, and treat abstention as an explicit operational outcome.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Machine Learning
Dream to Assist: Intent-Conditioned World Models for Shared Vehicle Control
Repulsive Population Policy Gradients for Behaviorally Diverse Reinforcement Learning
AdaTutor: Adaptive Tutoring through Pedagogical Strategy Selection
Property-Based Testing for Robust Softmax and Cross-Entropy
Video World Models Need Predictable Latents
Get the few papers worth your time in Machine Learning every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.