Junyi Li, Ji-Rong Wen, Wayne Zhao, Tianyi Tang, Jian-Yun Nie
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
We study the text generation task under the approach of pre-trained language models (PLMs). Typically, an auto-regressive (AR) method is adopted for generating texts in a token-bytoken manner. Despite many advantages of AR generation, it usually suffers from inefficient inference. Therefore, non-autoregressive (NAR) models are proposed to generate all target tokens simultaneously. However, NAR models usually generate texts of lower quality due to the absence of token dependency in the output text. In this paper, we propose ELMER: an Efficient and effective PLM for NAR tExt geneRation to explicitly model the token dependency during NAR generation. By leveraging the early exit technique, ELMER enables the token generations at different layers, according to their prediction confidence (a more confident token will exit at a lower layer). Besides, we propose a novel pre-training objective, Layer Permutation Language Modeling, to pre-train ELMER by permuting the exit layer for each token in sequences. Experiments on three text generation tasks show that ELMER significantly outperforms NAR models and further narrows the performance gap with AR PLMs (e.g., ELMER (29.92) vs BART (30.61) ROUGE-L in XSUM) while achieving over 10 times inference speedup.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2210.13304")
get_code_for_paper("2210.13304")
have("2210.13304")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2210.13304.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2210.13304)
Connect an agent — have() is free.