source: arxiv statistics ml: prediction-powered active testing
level: technical
active testing aims to estimate model risk with fewer labels by choosing which test points to annotate. existing methods do not use predictions from powerful black-box models, even though those predictions are often available. the new framework, prediction-powered active testing (ppat), combines the unbiased lure estimator with a control variate based on model predictions. instead of treating predictions as biased labels, ppat uses them to residualize the loss, which keeps the estimate unbiased but lowers its variance.
ppat also changes the acquisition strategy. the authors derive oracle and practical surrogate-based rules for selecting points that specifically reduce the variance of the ppat estimator. this means the system not only estimates risk more efficiently but also actively picks the most informative examples to label. the approach is designed for settings where labels are costly but model predictions are cheap, such as in medical imaging or large-scale content moderation.
experiments show that ppat achieves lower mean squared error than standard active testing and other baselines across various tasks. the method works with any black-box predictor and does not require retraining. it provides a simple way to integrate existing models into the evaluation pipeline, making risk estimation more label-efficient without sacrificing statistical guarantees.
why it matters: it enables more accurate model evaluation with fewer labeled examples, saving time and money in ai development and deployment.
source: arxiv statistics ml: prediction-powered active testing