level: research
Researchers introduced AutoSI, a framework that automates selective inference for algorithms written in NumPy-like code. Selective inference corrects p-values when the same data is used to choose and test a hypothesis, but existing methods require experts to manually derive the selection event for each algorithm. AutoSI constructs this event automatically from the algorithm’s operations, eliminating hand derivation and broadening the class of algorithms that can be handled.
The framework works by tracking the algorithm’s execution to build a piecewise-linear representation of the selection event. In experiments on feature selection, AutoSI produced valid p-values under the null hypothesis, with empirical rejection rates close to the nominal level. It currently supports algorithms with rationally expressible operations, meaning those that can be written as a finite sequence of linear transformations, max, min, and comparisons, which covers many common data science procedures.
Selective inference has been limited to a narrow set of algorithms because each new method required custom statistical work. AutoSI lowers this barrier, making exact post-selection inference accessible to practitioners who can code their algorithms in a familiar style. The approach could help validate discoveries in high-dimensional data analysis, where feature selection and model choice often introduce bias. The code is available on GitHub, and the paper includes theoretical guarantees for the method’s validity.
why it matters: AutoSI makes it practical to get statistically valid p-values after data-driven model selection, reducing false discoveries in AI and data science workflows.