knowledge distillation works by sparsifying interactions in llms
a unified framework reveals that various knowledge distillation methods for large language models share a common mechanism: they force student models to use fewer interactions while zeroing out the rest.
ai