level: research
pretrained classification models often fail when deployed on data that differs from their training distribution. this paper introduces audited conformal prediction (acp), a method that uses a small labeled sample from the target population to train an auxiliary audit model. the audit model identifies inputs where the legacy classifier is likely to make errors. by incorporating the audit model's outputs into the conformal prediction framework, acp produces prediction sets with guaranteed marginal coverage and better conditional coverage in practice.
the authors develop two integration strategies. the first aims for marginal coverage while improving conditional performance. the second provides explicit group-conditional coverage guarantees. both approaches come with theoretical backing. experiments on synthetic and real-world datasets show that acp outperforms existing methods, especially in achieving higher conditional coverage. the method works without retraining the original classifier, making it practical for deployed systems.
acp addresses a key challenge in machine learning: reliable uncertainty quantification under distribution shift. it is particularly useful when only limited target data is available. the audit model can be lightweight and trained quickly. the paper also discusses trade-offs between the two strategies, helping practitioners choose based on their needs. the approach is model-agnostic and can be applied to any pretrained classifier.
why it matters: it enables safer deployment of ai classifiers in changing environments by providing reliable prediction sets without retraining, using only a small amount of target data.