SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2107.00181 · CVPR · 2021

Revisiting Knowledge Distillation: An Inheritance and Exploration Framework

Houqiang Li, Xian-Sheng Hua, Jianqiang Huang, Tongliang Liu, Xinmei Tian, Zhen Huang, Xu Shen, Bing Deng, Jun Xing

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

in which a student model is split into two parts -inheritance and exploration. The inheritance part is learned with a similarity loss to transfer the existing learned knowledge from the teacher model to the student model, while the exploration part is encouraged to learn representations different from the inherited ones with a dis-similarity loss. Our IE-KD framework is generic and can be easily combined with existing distillation or mutual learning methods for training deep neural networks. Extensive experiments demonstrate that these two parts can jointly push the student model to learn more diversified and effective representations, and our IE-KD can be a general technique to improve the student network to achieve SOTA performance. Furthermore, by applying our IE-KD to the training of two networks, the performance of both can be improved w.r.t. deep mutual learning.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2107.00181")
get_code_for_paper("2107.00181")
have("2107.00181")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2107.00181.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2107.00181.svg)](https://syntology.ai/paper/2107.00181)

Connect an agent — have() is free.