Yi Yang, Chen Zhang, Dawei Song
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. If it links a repository, it is listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Recent advances in distilling pretrained language models have discovered that, besides the expressiveness of knowledge, the studentfriendliness should be taken into consideration to realize a truly knowledgeable teacher. Based on a pilot study, we find that overparameterized teachers can produce expressive yet student-unfriendly knowledge and are thus limited in overall knowledgeableness. To remove the parameters that result in studentunfriendliness, we propose a sparse teacher trick under the guidance of an overall knowledgeable score for each teacher parameter. The knowledgeable score is essentially an interpolation of the expressiveness and studentfriendliness scores. The aim is to ensure that the expressive parameters are retained while the student-unfriendly ones are removed. Extensive experiments on the GLUE benchmark show that the proposed sparse teachers can be dense with knowledge and lead to students with compelling performance in comparison with a series of competitive baselines. 1
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2210.03923")
get_code_for_paper("2210.03923")
have("2210.03923")
Connect an agent — have() is free.