SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2303.07457 · AAAI · 2023

AMOM: Adaptive Masking over Masking for Conditional Masked Language Model

Lijun Wu, Tao Qin, Tie-Yan Liu, Juntao Li, Min Zhang, Yisheng Xiao, Ruiyang Xu

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Transformer-based autoregressive (AR) methods have achieved appealing performance for varied sequence-tosequence generation tasks, e.g., neural machine translation, summarization, and code generation, but suffer from low inference efficiency. To speed up the inference stage, many non-autoregressive (NAR) strategies have been proposed in the past few years. Among them, the conditional masked language model (CMLM) is one of the most versatile frameworks, as it can support many different sequence generation scenarios and achieve very competitive performance on these tasks. In this paper, we further introduce a simple yet effective adaptive masking over masking strategy to enhance the refinement capability of the decoder and make the encoder optimization easier. Experiments on 3 different tasks (neural machine translation, summarization, and code generation) with 15 datasets in total confirm that our proposed simple method achieves significant performance improvement over the strong CMLM model. Surprisingly, our proposed model yields state-of-the-art performance on neural machine translation (34.62 BLEU on WMT16 EN→RO, 34.82 BLEU on WMT16 RO→EN, and 34.84 BLEU on IWSLT De→En) and even better performance than the AR Transformer on 7 benchmark datasets with at least 2.2× speedup. Our code is available at GitHub 1 .

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2303.07457")
get_code_for_paper("2303.07457")
have("2303.07457")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2303.07457.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2303.07457.svg)](https://syntology.ai/paper/2303.07457)

Connect an agent — have() is free.