SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2302.01530 · EACL Findings · 2023

Revisiting Intermediate Layer Distillation for Compressing Language Models: An Overfitting Perspective

Chang, Jongwoo Ko, Minchan Jeong, Sukjin Hong, Yun, Seungjoon Park, Euijai Kt, Ahn, Du-Seong Kt, Se-Young Kt

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Knowledge distillation (KD) is a highly promising method for mitigating the computational problems of pre-trained language models (PLMs). Among various KD approaches, Intermediate Layer Distillation (ILD) has been a de facto standard KD method with its performance efficacy in the NLP field. In this paper, we find that existing ILD methods are prone to overfitting to training datasets, although these methods transfer more information than the original KD. Next, we present the simple observations to mitigate the overfitting of ILD: distilling only the last Transformer layer and conducting ILD on supplementary tasks. Based on our two findings, we propose a simple yet effective consistency-regularized ILD (CR-ILD), which prevents the student model from overfitting the training dataset. Substantial experiments on distilling BERT on the GLUE benchmark and several synthetic datasets demonstrate that our proposed ILD method outperforms other KD techniques. Our code is available at https: //github.com/jongwooko/CR-ILD.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2302.01530")
get_code_for_paper("2302.01530")
have("2302.01530")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2302.01530.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2302.01530.svg)](https://syntology.ai/paper/2302.01530)

Connect an agent — have() is free.