Jie Tang, Weng Lam, Peng Zhang, Shu Zhao, Shicheng Tan, Yuanchun Wang, Wenwen Gong
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Currently, the reduction in the parameter scale of large-scale pre-trained language models (PLMs) through knowledge distillation has greatly facilitated their widespread deployment on various devices. However, the deployment of knowledge distillation systems faces great challenges in real-world industrial-strength applications, which require the use of complex distillation methods on even larger-scale PLMs (over 10B), limited by memory on GPUs and the switching of methods. To overcome these challenges, we propose GKD, a general knowledge distillation framework that supports distillation on larger-scale PLMs using various distillation methods. With GKD, developers can build larger distillation models on memorylimited GPUs and easily switch and combine different distillation methods within a single framework. Experimental results show that GKD can support the distillation of at least 100B-scale PLMs and 25 mainstream methods on 8 NVIDIA A100 (40GB) GPUs. 1
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2306.06629")
get_code_for_paper("2306.06629")
have("2306.06629")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2306.06629.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2306.06629)
Connect an agent — have() is free.