Hinrich Schütze, Barbara Plank, Xinpeng Wang, Leonie Weissweiler
We lifted 11 functions out of this paper's own repositories and ran 2 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| mainlp/How-to-distill-your-BERT | canonical | 1 of 10 |
| mainlp/how-to-distill-your-bert | canonical | 1 of 1 |
| Function | Status | Where it lives |
|---|---|---|
| get_valid_dataloader | Ran | mainlp/How-to-distill-your-BERT/task_agnostic_distillation/run_pretraining.py code served (permissive licence) · get_code("76a475fa13f37cec") |
| task_specific_config | Ran | mainlp/how-to-distill-your-bert/task_specific_distillation/experiments.py code served (permissive licence) · get_code("de41371cecbab67c") |
| att_kl | Not yet run | mainlp/How-to-distill-your-BERT/task_agnostic_distillation/methods/feature_distill.py code served (permissive licence) · get_code("36f7375b15089e34") |
| att_val_kl | Not yet run | mainlp/How-to-distill-your-BERT/task_agnostic_distillation/methods/feature_distill.py code served (permissive licence) · get_code("0888d8eb55277184") |
| bias_gelu | Not yet run | mainlp/How-to-distill-your-BERT/task_agnostic_distillation/pretraining/modeling.py code served (permissive licence) · get_code("f114723b758d8789") |
| cosine_similarity | Not yet run | mainlp/How-to-distill-your-BERT/task_agnostic_distillation/methods/pear_loss.py code served (permissive licence) · get_code("c5070fa84fd2b490") |
| data_aug | Not yet run | mainlp/How-to-distill-your-BERT/task_agnostic_distillation/methods/feature_distill.py code served (permissive licence) · get_code("e4c865c0ef8d62b4") |
| f_gelu | Not yet run | mainlp/How-to-distill-your-BERT/task_agnostic_distillation/pretraining/modeling.py code served (permissive licence) · get_code("2fd4f78b58e81a10") |
| get_deepspeed_config | Not yet run | mainlp/How-to-distill-your-BERT/task_agnostic_distillation/pretraining/modeling.py code served (permissive licence) · get_code("87b054b1860fbd9e") |
| inter_class_relation | Not yet run | mainlp/How-to-distill-your-BERT/task_agnostic_distillation/methods/pear_loss.py code served (permissive licence) · get_code("cda3d1271f46dae7") |
| pearson_correlation | Not yet run | mainlp/How-to-distill-your-BERT/task_agnostic_distillation/methods/pear_loss.py code served (permissive licence) · get_code("78c3c61943df0cc0") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Recently, various intermediate layer distillation (ILD) objectives have been shown to improve compression of BERT models via Knowledge Distillation (KD). However, a comprehensive evaluation of the objectives in both taskspecific and task-agnostic settings is lacking. To the best of our knowledge, this is the first work comprehensively evaluating distillation objectives in both settings. We show that attention transfer gives the best performance overall. We also study the impact of layer choice when initializing the student from the teacher layers, finding a significant impact on the performance in task-specific distillation. For vanilla KD and hidden states transfer, initialisation with lower layers of the teacher gives a considerable improvement over higher layers, especially on the task of QNLI (up to an absolute percentage change of 17.8 in accuracy). Attention transfer behaves consistently under different initialisation settings. We release our code as an efficient transformer-based model distillation framework for further studies. 1
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2305.15032")
get_code_for_paper("2305.15032")
have("2305.15032")
Connect an agent — have() is free.