arXiv preprint ·
Variance-adaptive confidence regions for fixed-benchmark binomial mixtures, with application to LLM evaluation
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
We construct confidence intervals for linear functionals $θ(F_m)=\int h\,dF_m$ of the empirical law $F_m$ of success probabilities $p_1,\dots,p_m$, from independent counts $c_i\sim\mathrm{Bin}(n,p_i)$, as for a language model run $n$ times per problem. Coverage is at least $1-α$ for every $n$, $m$ and fixed $(p_i)$, simultaneously for all bounded $h$. The variance-adaptive (VA) region tests each candidate mixing law with the conditional variance of the tail counts it implies. This variance is linear in the law, so the region is a linear program, and cell envelopes keep it valid off-grid. Regions calibrated for iid sampling target a super-population law and are conservative for $θ(F_m)$. With all $n$ tail counts, fixed $n$, continuous $h$ and a grid refined as $m\to\infty$, the identification part of VA's width is first-order worst-case optimal, as for any localization with the same index set. Under a rule fixed before the main comparator runs, VA was narrower than a two-stage plug-in band at every $n$ (median width ratio 0.60 at $n=128$), but not than the asymptotic, pointwise AMARI interval at $n=128$; on real rollout pools it was wider than both pointwise and Bonferroni-adjusted AMARI at $n\le128$. For $P(p>0)$ and pass@$k$ with $k\gg n$ the data support only lower bounds.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Methodology
Decorrelation Before Variable Selection: Left and Right Transforms and Their Trade-Off
COSI: A Framework for Robustly Valid Inference
FARCH: A Functional ARCH Model for Calibrated Prediction Bands
SOTA or Luck? The Winner's Curse in LLM Leaderboards
FAST: Functional Adaptive Sensing and Testing for Quickest Change Detection
Get the few papers worth your time in Methodology every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.