arXiv preprint ·
Kernel-based External Validity of Experimental Studies
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
The external validity of a study is the extent to which its findings generalize to unseen experimental conditions, such as datasets. We formalize the results of a study together with its research question, and define external validity as the probability that two independent and same-sized replications of the study produce results whose maximum mean discrepancy (MMD) is below a given tolerance. This approach, centered on the choice of a space of results and of an appropriate reproducing kernel for the MMD, is flexible enough to be deployed on multiple different types of results and research questions. Additionally, we establish an asymptotic power-law relation between the size of a study and the quantiles of the MMD, which we use to estimate the desired size of an externally valid study from a small-scale preliminary study. We deploy our framework to rankings with ties and discuss kernels suited to different research questions. Experiments on existing case studies and on a simulated newly designed study illustrate the applicability of the framework. We release our code as part of the genexpy Python module.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Methodology
Two-way Guidance: Simultaneous Learning of Summary Statistics for Generative Bayes
Prior-Informed Generalizable Low-Rank Representation Learning via Regularized Adversarial Reweighting
From Imbalance to Evidence: Deep Nonparametric Testing for Complex Data under Severe Sample Imbalance
Copula Active Subspaces II: Error Decomposition, A Posteriori Estimation, and Sharpness of the Bounds
Spatial Prediction with Guided Diffusion
Get the few papers worth your time in Methodology every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.