source: arxiv statistics ml: conformal prediction via transported beta laws
level: research
split conformal prediction gives finite-sample marginal coverage when data are exchangeable, but this guarantee averages over the random calibration set. the actual coverage for a fixed calibration set can vary. in the continuous i.i.d. case, the calibration-conditional coverage follows a beta distribution with parameters k and n+1-k, where k is the rank of the test score and n is the calibration size. the usual marginal coverage is just the mean of this beta law.
the authors treat this beta distribution as a reference object and measure deviations from it using wasserstein distances on the unit interval. this approach yields direct bounds on marginal coverage gaps and on the probability of bad calibration. it also separates different sources of non-i.i.d. behavior: test-side distribution shift acts through a transport map on the coverage scale, while dependence in the calibration data changes the order-statistic law itself.
the framework is demonstrated in scale-shift, clustered, and stationary mixing settings. by quantifying how far the actual coverage law is from the ideal beta, practitioners can assess the reliability of conformal prediction under various departures from exchangeability. the method provides a principled way to understand and control coverage errors beyond the standard marginal guarantee.
why it matters: this work gives data scientists a tool to measure and bound coverage errors in conformal prediction when data are not perfectly exchangeable, improving trust in uncertainty estimates for real-world applications.
source: arxiv statistics ml: conformal prediction via transported beta laws