SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2210.16732 · EMNLP · 2022

How Far are We from Robust Long Abstractive Summarization?

Shirui Pan, He Zhang, Ming Liu, Jiaxin Ju, Huan Koh

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Abstractive summarization has made tremendous progress in recent years. In this work, we perform fine-grained human annotations to evaluate long document abstractive summarization systems (i.e., models and metrics) with the aim of implementing them to generate reliable summaries. For long document abstractive models, we show that the constant strive for state-of-the-art ROUGE results can lead us to generate more relevant summaries but not factual ones. For long document evaluation metrics, human evaluation results show that ROUGE remains the best at evaluating the relevancy of a summary. It also reveals important limitations of factuality metrics in detecting different types of factual errors and the reasons behind the effectiveness of BARTScore. We then suggest promising directions in the endeavor of developing factual consistency metrics. Finally, we release our annotated long document dataset with the hope that it can contribute to the development of metrics across a broader range of summarization settings.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2210.16732")
get_code_for_paper("2210.16732")
have("2210.16732")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2210.16732.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2210.16732.svg)](https://syntology.ai/paper/2210.16732)

Connect an agent — have() is free.