SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2305.13477 · EMNLP · 2023

Look-back Decoding for Open-Ended Text Generation

Xuezhe Ma, Chunting Zhou, Asli Celikyilmaz, Nan Xu

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Given a prefix (context), open-ended generation aims to decode texts that are coherent, which do not abruptly drift from previous topics, and informative, which do not suffer from undesired repetitions. In this paper, we propose Look-back , an improved decoding algorithm that leverages the Kullback-Leibler divergence to track the distribution distance between current and historical decoding steps. Thus Lookback can automatically predict potential repetitive phrase and topic drift, and remove tokens that may cause the failure modes, restricting the next token probability distribution within a plausible distance to the history. We perform decoding experiments on document continuation and story generation, and demonstrate that Look-back is able to generate more fluent and coherent text, outperforming other strong decoding methods significantly in both automatic and human evaluations 1 . Prefix: In addition to live broadcasts FIFA Fan Fests offer food and beverages, merchandise and various entertainment events by local and international artists. The start of 2006 World Cup was Repetitive Continuation S1: (GPT2-XL greedy) marked by the FIFA Fan Fest in Sao Paulo, Brazil.\n\n The FIFA Fan Fest is a unique event that brings together fans from all over the world to celebrate the FIFA World Cup . The FIFA Fan Fest is a unique event that brings together fans from... S2: (davinci-002 greedy) celebrated with a concert by Bon Jovi in Berlin.\n\n The FIFA Fan Fest in Munich was located at the Olympic Park.\n\n The FIFA Fan Fest in Frankfurt was located at the Römerberg.\n\n The FIFA Fan Fest in Hamburg was located at the ... Prefix: Burkan died from an attack of acute indigestion at his country home in Great Neck on June 6, 1936. Three thousand people attended his funeral at Off-Topic Continuation S3: (ada-001 greedy) the local church.\n\n The authorities are investigating how Khedira managed to enter the house and what role he played in the attack.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2305.13477")
get_code_for_paper("2305.13477")
have("2305.13477")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2305.13477.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2305.13477.svg)](https://syntology.ai/paper/2305.13477)

Connect an agent — have() is free.