We lifted 27 functions out of this paper's own repositories and ran 18 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| Daniel-H-99/Patient-Knowledge-Distillation | pwc_unofficial | 13 of 17 |
| eunanomist/PKD_BERT | pwc_unofficial | 5 of 10 |
| Function | Status | Where it lives |
|---|---|---|
| swish | Ran | Daniel-H-99/Patient-Knowledge-Distillation/BERT/pytorch_pretrained_bert/modeling.py pointer only (licence: MIT) · get_code("0f786c407fb1ee4c") |
| acc_and_f1 | Ran | Daniel-H-99/Patient-Knowledge-Distillation/src/nli_data_processing.py pointer only (licence: MIT) · get_code("cac113ca87b9d9f3") |
| boolean_string | Ran | Daniel-H-99/Patient-Knowledge-Distillation/src/argument_parser.py pointer only (licence: MIT) · get_code("b1f9cebf0d177140") |
| build_tf_to_pytorch_map | Ran | eunanomist/PKD_BERT/BERT/pytorch_pretrained_bert/modeling_transfo_xl.py pointer only (licence: MIT) · get_code("35ef111fc00c21b4") |
| convert_examples_to_features | Ran | Daniel-H-99/Patient-Knowledge-Distillation/src/nli_data_processing.py pointer only (licence: MIT) · get_code("379ade5858acdbfa") |
| convert_examples_to_features | Ran | Daniel-H-99/Patient-Knowledge-Distillation/src/race_data_processing.py pointer only (licence: MIT) · get_code("b30ee2907e5cd3a6") |
| count_parameters | Ran | Daniel-H-99/Patient-Knowledge-Distillation/src/utils.py pointer only (licence: MIT) · get_code("3ea89c558af6c4b4") |
| distillation_loss | Ran | Daniel-H-99/Patient-Knowledge-Distillation/src/KD_loss.py pointer only (licence: MIT) · get_code("5ff7ab3a0c0d6da0") |
| gelu | Ran | Daniel-H-99/Patient-Knowledge-Distillation/BERT/pytorch_pretrained_bert/modeling_gpt2.py pointer only (licence: MIT) · get_code("8d23fbe2b99b840b") |
| gelu | Ran | Daniel-H-99/Patient-Knowledge-Distillation/BERT/pytorch_pretrained_bert/modeling.py pointer only (licence: MIT) · get_code("fdc64f4c72036ae4") |
| is_folder_empty | Ran | Daniel-H-99/Patient-Knowledge-Distillation/src/argument_parser.py pointer only (licence: MIT) · get_code("615f129ea793785e") |
| load_model | Ran | Daniel-H-99/Patient-Knowledge-Distillation/src/utils.py pointer only (licence: MIT) · get_code("9e42bf8972501658") |
| patience_loss | Ran | Daniel-H-99/Patient-Knowledge-Distillation/src/KD_loss.py pointer only (licence: MIT) · get_code("16afb11eb02e8e88") |
| read_mrc_examples | Ran | Daniel-H-99/Patient-Knowledge-Distillation/src/race_data_processing.py pointer only (licence: MIT) · get_code("28d61d48898d30c4") |
| sample_logits | Ran | eunanomist/PKD_BERT/BERT/pytorch_pretrained_bert/modeling_transfo_xl_utilities.py pointer only (licence: MIT) · get_code("93228a3eb5c4d179") |
| url_to_filename | Ran | eunanomist/PKD_BERT/BERT/pytorch_pretrained_bert/file_utils.py pointer only (licence: MIT) · get_code("5eff22fa0a651276") |
| warmup_cosine | Ran | eunanomist/PKD_BERT/BERT/pytorch_pretrained_bert/optimization.py pointer only (licence: MIT) · get_code("007a8d2955fb555d") |
| warmup_linear | Ran | eunanomist/PKD_BERT/BERT/pytorch_pretrained_bert/optimization.py pointer only (licence: MIT) · get_code("e0960afdb0d64aa7") |
| cached_path | Not yet run | eunanomist/PKD_BERT/BERT/pytorch_pretrained_bert/file_utils.py pointer only (licence: MIT) · get_code("df9f10631fa5d322") |
| filename_to_url | Not yet run | eunanomist/PKD_BERT/BERT/pytorch_pretrained_bert/file_utils.py pointer only (licence: MIT) · get_code("6db16fe8f67e56b6") |
| fill_tensor | Not yet run | Daniel-H-99/Patient-Knowledge-Distillation/src/utils.py pointer only (licence: MIT) · get_code("01803af97cd5feef") |
| load_tf_weights_in_bert | Not yet run | Daniel-H-99/Patient-Knowledge-Distillation/BERT/pytorch_pretrained_bert/modeling.py pointer only (licence: MIT) · get_code("b1a7180868dd8d6a") |
| load_tf_weights_in_gpt2 | Not yet run | Daniel-H-99/Patient-Knowledge-Distillation/BERT/pytorch_pretrained_bert/modeling_gpt2.py pointer only (licence: MIT) · get_code("a62604f618a8cc82") |
| load_tf_weights_in_openai_gpt | Not yet run | eunanomist/PKD_BERT/BERT/pytorch_pretrained_bert/modeling_openai.py pointer only (licence: MIT) · get_code("46f305675277e757") |
| load_tf_weights_in_transfo_xl | Not yet run | eunanomist/PKD_BERT/BERT/pytorch_pretrained_bert/modeling_transfo_xl.py pointer only (licence: MIT) · get_code("1ebc2bf2c3cddc4d") |
| simple_accuracy | Not yet run | Daniel-H-99/Patient-Knowledge-Distillation/src/nli_data_processing.py pointer only (licence: MIT) · get_code("3c241ecfe3749a6d") |
| warmup_constant | Not yet run | eunanomist/PKD_BERT/BERT/pytorch_pretrained_bert/optimization.py pointer only (licence: MIT) · get_code("59e4e730a9dc5dc6") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Pre-trained language models such as BERT have proven to be highly effective for natural language processing (NLP) tasks. However, the high demand for computing resources in training such models hinders their application in practice. In order to alleviate this resource hunger in large-scale model training, we propose a Patient Knowledge Distillation approach to compress an original large model (teacher) into an equally-effective lightweight shallow network (student). Different from previous knowledge distillation methods, which only use the output from the last layer of the teacher network for distillation, our student model patiently learns from multiple intermediate layers of the teacher model for incremental knowledge extraction, following two strategies: ($i$) PKD-Last: learning from the last $k$ layers; and ($ii$) PKD-Skip: learning from every $k$ layers. These two patient distillation schemes enable the exploitation of rich information in the teacher's hidden layers, and encourage the student model to patiently learn from and imitate the teacher through a multi-layer distillation process. Empirically, this translates into improved results on multiple NLP tasks with significant gain in training efficiency, without sacrificing model accuracy.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("1908.09355")
get_code_for_paper("1908.09355")
have("1908.09355")
Connect an agent — have() is free.