source: arxiv statistics ml: optimal self-distillation for rectified flow via linear probing
level: research
modern generative models often learn from data created by other models, which can help or hurt performance. this work looks at self-distillation for rectified flow, a type of generative model. the goal is to train a student model using a mix of true velocities and velocities from a suboptimal teacher. the researchers prove that for a simplified linear setting with ridge regularization, there is an exact formula for the best mixing ratio.
the optimal mixing coefficient follows a sign rule. if the teacher is under-regularized, meaning it fits noise too closely, the student should use positive mixing to add more real data. if the teacher is over-regularized, meaning it is too smooth, the student should use negative mixing, which effectively pushes away from the teacher's predictions. this rule guarantees the student will always outperform the teacher in terms of velocity prediction error, as long as the teacher is not already optimal along the regularization path.
the paper also introduces a practical way to find the best mixing weight without expensive grid search. they propose a one-shot generalized cross-validation method and a validation-based tuning procedure. these techniques estimate the optimal coefficient directly from the data, making the approach feasible for real-world use. the findings provide a theoretical foundation for self-distillation in rectified flow and suggest simple strategies for improving generative models.
why it matters: this gives a principled way to improve generative models by reusing their own outputs, with a clear rule for when to trust or ignore the teacher.
source: arxiv statistics ml: optimal self-distillation for rectified flow via linear probing