How Well Do Prior-Data Fitted Networks Amortize Bayesian Inference?

Abstract

Prior-data fitted networks (PFNs) are trained to approximate the Bayesian posterior predictive distribution (PPD) of a synthetic prior, yet results about the approximation quality are surprisingly thin. Taking an empirical risk minimization perspective, we decompose the deployment error of PFNs into prior, context, and amortization error, and measure the latter against a reference PPD under a realistic prior. Results show that PFNs generally recover the targeted PPD very well. Nevertheless, we find that parts of the residual amortization error persist, especially in \textit{rare} settings under the prior pretraining distribution, even when scaling the model size, which manifests in disagreement between independently pretrained PFNs and leaves uncertainty about the predictor at inference time. As a remedy, we study multiple types of Bayesian model averages and find that these can consistently reduce this uncertainty and improve predictions up to full-size TabICLv2 models.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Machine Learning (Statistics)

• arXiv preprint · 9 October 2026

  Scalable Learning of Determinantal Point Processes with Kernel-Structured Interactions

  Takahiro Kawashima and Hideitsu Hino
• arXiv preprint · 10 October 2026

  Prior-Informed Generalizable Low-Rank Representation Learning via Regularized Adversarial Reweighting

  Keyao Zhan, Junwei Lu and Tianxi Cai
• arXiv preprint · 9 October 2026

  Two-way Guidance: Simultaneous Learning of Summary Statistics for Generative Bayes

  Percy S. Zhai, Max Pachman, Peverel D. Shipley and 1 other
• arXiv preprint · 10 October 2026

  ByteMark: A Unified Byte-Level Watermark for Detection Across Language Models

  Xiaopu Wang and Runze Li
• arXiv preprint · 9 October 2026

  Universal Bounds for Self-Normalized Kullback-Leibler Estimation

  Buu Phan, Lester Mackey and David Alvarez-Melis

All new papers in Machine Learning (Statistics)