arXiv preprint ·
Software Testing Meets Bayesian Workflow: A Literature-Driven Artifact Study of Probabilistic Programs
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Automated testing is an effective software quality assurance method, but it is rarely used for probabilistic models. This is despite the heavy use of probabilistic programming for data analytics in research and engineering, where the consequences of errors are dire. A key obstacle is the specification of testable correctness properties for probabilistic models, known as the oracle problem. Like machine learning, probabilistic programming is used precisely where no analytical specification exists: had we been able to write one, we would have written an algorithm instead of fitting a model. An alternative is partial, proxy correctness specifications. The statistics literature already offers many, but as quality guidelines that require human judgment: comparing plots, visually analyzing shapes of distributions, assessing whether parameter values are ''far'' or ''close enough''. These guidelines are not systematically automated. In this work, we elicit, implement, and study testable correctness properties of probabilistic models. We extract 17 broadly applicable properties for probabilistic models from one of the most widely used textbooks in Bayesian data analysis, Statistical Rethinking by McElreath. For each property, we select and implement a test oracle, and study its utility for automatic testing, using a population of 59 probabilistic models from academic and industrial sources. We find realistic use cases (models under test) for all the properties. We observe that the vast majority of tests are not flaky in practice, despite the theoretical potential for flakiness. To the best of our knowledge, we are the first to investigate the effectiveness of mutation testing for probabilistic models, showing that standard mutation operators do not capture faults tracked by the probabilistic properties.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Software Engineering
When Agent Evaluation Becomes Unreliable: An Empirical Study of Oracle Failures and Configuration-Dependent Measurement in an LLM Agent Benchmark
Cross-Project Bug Localization: What Works, What Doesn't, and Why
SafeDSE: Joint Exploration of Safety Architecture Patterns, Mapping, and Scheduling for Dataflow Applications With Safety Arguments (Extended Version)
StateHarness: Maintaining Evolving Project State for Long-Horizon Coding Agents
EvoWeb: A Benchmark for Coding Agents on Evolving Web Applications
Get the few papers worth your time in Software Engineering every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.