Even your Teacher Needs Guidance: Ground-Truth Targets Dampen Regularization Imposed by Self-Distillation

Kenneth Borup, Lars N. Andersen

Research output: Contribution to conferencePaperResearchpeer-review

72 Downloads (Pure)

Abstract

Knowledge distillation is classically a procedure where a neural network is trained on the output of another network along with the original targets in order to transfer knowledge between the architectures. The special case of self-distillation, where the network architectures are identical, has been observed to improve generalization accuracy. In this paper, we consider an iterative variant of self-distillation in a kernel regression setting, in which successive steps incorporate both model outputs and the ground-truth targets. This allows us to provide the first theoretical results on the importance of using the weighted ground-truth targets in self-distillation. Our focus is on fitting nonlinear functions to training data with a weighted mean square error objective function suitable for distillation, subject to $\ell_2$ regularization of the model parameters. We show that any such function obtained with self-distillation can be calculated directly as a function of the initial fit, and that infinite distillation steps yields the same optimization problem as the original with amplified regularization. Finally, we examine empirically, both in a regression setting and with ResNet networks, how the choice of weighting parameter influences the generalization performance after self-distillation.
Original languageEnglish
Publication dateDec 2021
Publication statusPublished - Dec 2021
EventConference on Neural Information Processing Systems 2021 -
Duration: 6 Dec 202114 Dec 2021
https://nips.cc/

Conference

ConferenceConference on Neural Information Processing Systems 2021
Period06/12/202114/12/2021
Internet address

Fingerprint

Dive into the research topics of 'Even your Teacher Needs Guidance: Ground-Truth Targets Dampen Regularization Imposed by Self-Distillation'. Together they form a unique fingerprint.

Cite this