Calibration Assessment for Nested Uncertainty Sets

Abstract

We present a Bayesian framework for assessing the calibration of nested uncertainty sets from simulation studies. Calibration is typically evaluated by repeatedly simulating data from a known model and comparing empirical coverage probabilities with their nominal values, an approach that often requires hundreds or thousands of independent model evaluations to obtain reliable estimates. Such computational demands can be prohibitive when the forward models are expensive to evaluate, for example in data assimilation using a large Earth-system model, or may not be applicable, for exmaple when the output of the simulation is a single or multiple uncertainty sets rather than a full distribution. We present a method that leverages the nested structure of uncertainty sets across nominal coverage levels to assess the calibration of these models. The method, which is based on a Dirichlet-Multinomial model, allows for testing both exact and region-based notions of model calibration in an efficient manner using Bayes factors. Unlike methods that require a full predictive distribution, this method works for procedures that directly produce prediction or confidence intervals. We demonstrate the method on a data assimilation problem containing model form error and show that it is capable of detecting miscalibration even when there are only a limited number of simulation studies available.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Methodology

• arXiv preprint · 3 October 2026

  Neyman-Pearson Classification with Indecisions: Simultaneous Control of Selective Type I and Type II Errors

  Bradley Rava and Bowen Gang
• arXiv preprint · 4 October 2026

  Same Predictions, Different Harms: Causal Auditing of Patient World Models

  Yicheng Qi and Xiyi Xiong
• arXiv preprint · 4 October 2026

  Causal Lag Structure Discovery in Confounded Time Series via Orthogonalized Adaptive Estimation

  Hong Kiat Tan, Isaac-Neil Zanoria, James Chen and 2 others
• arXiv preprint · 2 October 2026

  Jumping up and down: Denoiser diffusion models for discrete ordinal data

  Yair Shenfeld, Ricardo Baptista and Stefano Peluchetti
• arXiv preprint · 5 October 2026

  Errors of LLM-Assisted Literature Retrieval in Environmental Science: A Comparison Study of Abstract versus Full-text Based Prompts

  Yanjun Chen, Yongfeng Zhang and Lanjing Zhang

All new papers in Methodology