level: research
calibration is often measured across all predictions, but the most harmful mistakes happen when a model is very confident yet wrong. this study looks at false-confidence concentration, meaning confident errors that bunch together in small, findable parts of the prediction space. the authors present falcon-discover, a post-hoc framework that works with any model. it scores each prediction using signals like confidence, how similar nearby predictions are, agreement with neighbors, and stability under small changes to the input.
experiments used seven binary tabular datasets, four random seeds, and five-fold cross-fitting. strong models like xgboost and catboost were tested. the results show that false-confidence concentration appears often but depends on the data and model setting. at the main confidence cutoff, ranking predictions by discrepancy signals did much better than the best calibration or trust-scoring baseline chosen from validation data. raw confidence scores alone found very few of these concentrated errors.
the work highlights that global calibration metrics can hide dangerous local error patterns. by focusing on where confident mistakes cluster, practitioners can spot risky prediction regions that need extra review. the method is model-agnostic and does not require retraining, making it easy to add to existing pipelines. it gives a practical way to audit model trustworthiness beyond average calibration scores.
why it matters: it helps data scientists find hidden pockets of overconfident wrong predictions that global metrics miss, improving model safety in high-stakes applications.