SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2102.13088 · NeurIPS · 2021

Even your Teacher Needs Guidance: Ground-Truth Targets Dampen Regularization Imposed by Self-Distillation

Kenneth Borup, Lars Andersen

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Knowledge distillation is classically a procedure where a neural network is trained on the output of another network along with the original targets in order to transfer knowledge between the architectures. The special case of self-distillation, where the network architectures are identical, has been observed to improve generalization accuracy. In this paper, we consider an iterative variant of self-distillation in a kernel regression setting, in which successive steps incorporate both model outputs and the ground-truth targets. This allows us to provide the first theoretical results on the importance of using the weighted ground-truth targets in self-distillation. Our focus is on fitting nonlinear functions to training data with a weighted mean square error objective function suitable for distillation, subject to 2 regularization of the model parameters. We show that any such function obtained with self-distillation can be calculated directly as a function of the initial fit, and that infinite distillation steps yields the same optimization problem as the original with amplified regularization. Furthermore, we provide a closed form solution for the optimal choice of weighting parameter at each step, and show how to efficiently estimate this weighting parameter for deep learning and significantly reduce the computational requirements compared to a grid search. * An updated version of this paper is published under the same title at the 35th Conference on Neural Information Processing Systems (NeurIPS 2021). 2 Often knowledge distillation is also referred to under the name Teacher-Student learning. 3 We will refer to the weighted outputs of the penultimate layer, i.e. pre-activation of the last layer, as logits. 4 See Section 2 for a brief overview, or see Wang and Yoon (2020) for a more exhaustive survey

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2102.13088")
get_code_for_paper("2102.13088")
have("2102.13088")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2102.13088.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2102.13088.svg)](https://syntology.ai/paper/2102.13088)

Connect an agent — have() is free.