level: research
researchers decomposed the output scores of large language models into sums of interactions, where each interaction captures a nonlinear relationship among a set of input variables like words. this decomposition allowed them to compare teacher and student models directly. they found that across different distillation techniques, student models consistently retained fewer interactions than their teachers. the suppressed interactions were effectively reduced to zero, meaning the student relies on a sparse subset of the original interactions.
the study shows that this sparsification of interactions is the common mechanism behind various knowledge distillation methods. regardless of the specific distillation approach, the student model learns to ignore many of the interactions present in the teacher. this simplification helps explain why distilled models can be smaller and faster while still performing well. the sparsification process appears to be a natural outcome of the distillation objective, not a side effect of any single method.
performance differences among distillation methods stem from how well each handles complex interactions. methods that better preserve or approximate high-order interactions—those involving many input variables—tend to produce stronger student models. simpler methods may struggle with these complex relationships, leading to worse performance. this insight suggests that future distillation techniques could focus on improving the transfer of complex interactions to boost student model quality.
why it matters: understanding that distillation sparsifies interactions can guide the design of more efficient and accurate compressed language models, directly impacting deployment in resource-limited settings.