SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2310.14971 · EMNLP · 2023

Penalty Decoding: Well Suppress the Self-Reinforcement Effect in Open-Ended Text Generation

Rui Wang, Wenhong Zhu, Hongkun Hao

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

The decoding algorithm is critical for openended text generation, transforming latent representations into coherent and meaningful outputs. This paper investigates the selfreinforcement effect in text generation and the effectiveness of a repetition penalty to mitigate it. However, determining the optimal repetition penalty value is challenging. To tackle this, we propose a forgetting mechanism that disregards distant tokens, reducing the burden of penalty selection. In addition, we introduce a length penalty to address overly short sentences caused by excessive penalties. Our penalty decoding approach incorporating three strategies helps resolve issues with sampling methods deviating from factual information. Experimental results demonstrate the efficacy of our approach in generating high-quality sentences resembling human output. 1

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2310.14971")
get_code_for_paper("2310.14971")
have("2310.14971")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2310.14971.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2310.14971.svg)](https://syntology.ai/paper/2310.14971)

Connect an agent — have() is free.