source: arxiv statistics ml: scorestop: gradient-based early stopping using functional score tests
level: technical
gradient boosted decision trees need a stopping rule to prevent overfitting. the usual approach watches a validation loss and stops if it does not improve for a set patience period. but the patience parameter has no clear meaning, and validation losses can be noisy or come from a user-defined gradient. we introduce scorestop, a gradient-based early-stopping rule. at each iteration, it tests whether the current model is the population risk minimizer. it uses a functional score test on validation data. the test statistic does not depend on the scale of the update direction and has a known distribution under the null hypothesis.
because the test uses gradients instead of loss values, it works for implicit losses like lambdarank and data-dependent losses like cox regression via influence functions. this makes it more flexible than loss-based stopping. the method does not require tuning a patience parameter, which simplifies training. the test automatically decides when further iterations are unlikely to improve the model.
experiments on synthetic and real data show that scorestop matches or beats standard early stopping. it often stops earlier without hurting accuracy. the approach is easy to add to existing gradient boosting frameworks. it provides a principled, statistical way to halt training, reducing the risk of overfitting while saving compute time.
why it matters: it gives a principled, automatic way to stop training gradient boosted trees, removing the need to tune a patience parameter and working for a wider range of loss functions.
source: arxiv statistics ml: scorestop: gradient-based early stopping using functional score tests