# Kenneth Borup

## Even your Teacher Needs Guidance: Ground-Truth Targets Dampen Regularization Imposed by Self-Distillation

Research output: Contribution to conferencePaperResearchpeer-review

### Documents

Knowledge distillation is classically a procedure where a neural network is trained on the output of another network along with the original targets in order to transfer knowledge between the architectures. The special case of self-distillation, where the network architectures are identical, has been observed to improve generalization accuracy. In this paper, we consider an iterative variant of self-distillation in a kernel regression setting, in which successive steps incorporate both model outputs and the ground-truth targets. This allows us to provide the first theoretical results on the importance of using the weighted ground-truth targets in self-distillation. Our focus is on fitting nonlinear functions to training data with a weighted mean square error objective function suitable for distillation, subject to $\ell_2$ regularization of the model parameters. We show that any such function obtained with self-distillation can be calculated directly as a function of the initial fit, and that infinite distillation steps yields the same optimization problem as the original with amplified regularization. Finally, we examine empirically, both in a regression setting and with ResNet networks, how the choice of weighting parameter influences the generalization performance after self-distillation.
Original language English Dec 2021 Published - Dec 2021 Conference on Neural Information Processing Systems 2021 - Duration: 6 Dec 2021 → 14 Dec 2021https://nips.cc/

### Conference

Conference Conference on Neural Information Processing Systems 2021 06/12/2021 → 14/12/2021 https://nips.cc/

### Bibliographical note

18 pages, 13 figures

Citationformats