We lifted 3 functions out of this paper's own repositories and ran 3 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| copy not recorded | — | 3 of 3 |
| Function | Status | Where it lives |
|---|---|---|
| eval_epoch | Ran | this paper's copy was not recorded; identical code first harvested from stdereka/knowledge-distillation pointer only · get_code("7308ca9b35a3b212") |
| fit_epoch | Ran | this paper's copy was not recorded; identical code first harvested from stdereka/knowledge-distillation pointer only · get_code("62f639fec5cb08e5") |
| train | Ran | this paper's copy was not recorded; identical code first harvested from stdereka/knowledge-distillation pointer only · get_code("00a58257f6886584") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Knowledge Distillation (KD) aims at improving the performance of a low-capacity student model by inheriting knowledge from a high-capacity teacher model. Previous KD methods typically train a student by minimizing a task-related loss and the KD loss simultaneously, using a pre-defined loss weight to balance these two terms. In this work, we propose to first transfer the backbone knowledge from a teacher to the student, and then only learn the task-head of the student network. Such a decomposition of the training process circumvents the need of choosing an appropriate loss weight, which is often difficult in practice, and thus makes it easier to apply to different datasets and tasks. Importantly, the decomposition permits the core of our method, Stage-by-Stage Knowledge Distillation (SSKD), which facilitates progressive feature mimicking from teacher to student. Extensive experiments on CIFAR-100 and ImageNet suggest that SSKD significantly narrows down the performance gap between student and teacher, outperforming state-of-the-art approaches. We also demonstrate the generalization ability of SSKD on other challenging benchmarks, including face recognition on IJB-A dataset as well as object detection on COCO dataset.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("1812.01819")
get_code_for_paper("1812.01819")
have("1812.01819")
Connect an agent — have() is free.