Yu Li, Chun Yuan, Zhendong Yang, Ailing Zeng, Zhe Li, Tianke Zhang
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Knowledge Distillation (KD) uses the teacher's logits as soft labels to guide the student, while self-KD does not need a real teacher to require the soft labels. This work unifies the formulations of the two tasks by decomposing and reorganizing the generic KD loss into a Normalized KD (NKD) loss and customized soft labels for both target class (image's category) and non-target classes named Universal Self-KD (USKD). We decompose the KD loss and find the non-target loss from it forces the student's non-target logits to match the teacher's, but the sum of the two nontarget logits is different, preventing them from being identical. NKD normalizes the non-target logits to equalize their sum. It can be generally used for KD and self-KD to better use the soft labels for distillation. USKD generates customized soft labels for both target and non-target classes without a teacher. It smooths the target logit of the student as the soft target label and uses the rank of the intermediate feature to generate the soft non-target labels with Zipf's law. For KD with teachers, NKD achieves state-of-the-art performance on CIFAR-100 and ImageNet, boosting the Im-ageNet Top-1 accuracy of Res-18 from 69.90% to 71.96% with a Res-34 teacher. For self-KD without teachers, USKD is the first method that can be effectively applied to both CNN and ViT models with negligible additional time and memory cost, resulting in new state-of-the-art results, such as 1.17% and 0.55% accuracy gains on ImageNet for Mo-bileNet and DeiT-Tiny, respectively. Code is available at https://github.com/yzd-v/cls_KD.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2303.13005")
get_code_for_paper("2303.13005")
have("2303.13005")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2303.13005.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2303.13005)
Connect an agent — have() is free.