Kyunghyun Cho, Richard Pang, Sean Welleck, Ilia Kulikov, Jaedeok Kim
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Despite strong performance on a variety of tasks, neural sequence models trained with maximum likelihood have been shown to exhibit issues such as length bias and degenerate repetition. We study the related issue of receiving infinite-length sequences from a recurrent language model when using common decoding algorithms. To analyze this issue, we first define inconsistency of a decoding algorithm, meaning that the algorithm can yield an infinite-length sequence that has zero probability under the model. We prove that commonly used incomplete decoding algorithms -greedy search, beam search, top-k sampling, and nucleus sampling -are inconsistent, despite the fact that recurrent language models are trained to produce sequences of finite length. Based on these insights, we propose two remedies which address inconsistency: consistent variants of top-k and nucleus sampling, and a selfterminating recurrent language model. Empirical results show that inconsistency occurs in practice, and that the proposed methods prevent inconsistency.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2002.02492")
get_code_for_paper("2002.02492")
have("2002.02492")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2002.02492.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2002.02492)
Connect an agent — have() is free.