level: research
when evaluating ai models on benchmarks, the test examples are often treated as independent, but they are not. they come from a shared pool, making them exchangeable. this paper studies functions of infinitely exchangeable random variables, which model such data. by conditioning on a latent directing measure, the total deviation of a function splits into two parts: a sampling fluctuation and a mixture fluctuation. the mixture part captures variation from the unknown data-generating distribution.
the authors derive a concentration inequality with an effective variance that combines the sum of squared bounded-difference constants and the subgaussian variance of the mixture. for zero-sum linear contrasts, like comparing a subsample mean to the full population mean, the mixture term cancels out exactly. this gives a tight, mixture-free hoeffding-type bound. the result explains why finite-exchangeable bounds tighten as the sample grows, linking to de finetti's theorem.
this framework is applied to ai benchmark uncertainty. when comparing two models on a fixed test set, the score difference is a zero-sum contrast. the new bound provides a sharper confidence interval than standard methods that ignore exchangeability. it helps practitioners decide if a measured improvement is statistically meaningful, without needing to assume independent test examples. the bound is simple to compute and does not require estimating the latent mixture variance.
why it matters: it gives data scientists a practical, tighter confidence interval for comparing ai model scores on benchmarks, reducing false claims of improvement.