Distilling the Knowledge in a Neural Network

Geoffrey Hinton, Oriol Vinyals, Jeff Dean

arXiv · PDF · Open in the Atlas

Code that ran

We lifted 37 functions out of this paper's own repositories and ran 16 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.

Repositories linked to this paper

• Adlik/model_optimizer
• DunZhang/KnowledgeDistillation
• Irtza/Keras_model_compression
• JunzWu/Distilling-the-Knowledge-in-a-Neural-Network
• KaiyuYue/mgd
• KellyYutongHe/Knowledge-Distillation-Net-with-Swish
• MasLiang/Learning-without-Forgetting-using-Pytorch
• MinJunKang/CIFAR-100--Knowledge-Distillation-with-Augmented-Data
• MinJunKang/ImageNet-Knowledge-Distillation
• MinJunKang/Knowledge-Distillation_MNIST
• NeelayS/Knowledge-Distillation
• TakieddineSOUALHI/Transfer_learning
• altndrr/persona
• ankur-98/Multi-domain-Knowledge-Distillation
• aquvitae/aquvitae
• aryanasadianuoit/distillation_deep
• chumingqian/Model_Compression_For_YOLOV3-V4
• elasticdeeplearning/edl
• franknb/Text-Summarization
• generation21/generation6011
• haro-cyber/KnowledgeDistillation
• haysacks/spot-the-coin
• hunto/image_classification_sota
• jaychoi12/LG_KD
• jchen98/compression
• jpmcd/TensorFlow-KnowledgeDistillation
• kaiyangzhou/on-device-dg
• kaung-htet-myat/Multi-teachers-Knowledge-Distillation
• knotgrass/Knowledge-Distillation
• lingyucai/Distilling-the-Knowledge-in-a-Neural-Network

More are linked than shown here.

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions [3]. Unfortunately, making predictions using a whole ensemble of models is cumbersome and may be too computationally expensive to allow deployment to a large number of users, especially if the individual models are large neural nets. Caruana and his collaborators [1] have shown that it is possible to compress the knowledge in an ensemble into a single model which is much easier to deploy and we develop this approach further using a different compression technique. We achieve some surprising results on MNIST and we show that we can significantly improve the acoustic model of a heavily used commercial system by distilling the knowledge in an ensemble of models into a single model. We also introduce a new type of ensemble composed of one or more full models and many specialist models which learn to distinguish fine-grained classes that the full models confuse. Unlike a mixture of experts, these specialist models can be trained rapidly and in parallel.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("1503.02531")
get_code_for_paper("1503.02531")
have("1503.02531")

Connect an agent — have() is free.