SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2205.09641 · EMNLP · 2022

SNAC: Coherence Error Detection for Narrative Summarization

Greg Durrett, Junyi Li, Tanya Goyal

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Progress in summarizing long texts is inhibited by the lack of appropriate evaluation frameworks. A long summary that appropriately covers the facets of that text must also present a coherent narrative, but current automatic and human evaluation methods fail to identify gaps in coherence. In this work, we introduce SNAC, a narrative coherence evaluation framework for fine-grained annotations of long summaries. We develop a taxonomy of coherence errors in generated narrative summaries and collect spanlevel annotations for 6.6k sentences across 150 book and movie summaries. Our work provides the first characterization of coherence errors generated by state-of-the-art summarization models and a protocol for eliciting coherence judgments from crowdworkers. Furthermore, we show that the collected annotations allow us to benchmark past work in coherence modeling and train a strong classifier for automatically localizing coherence errors in generated summaries. Finally, our SNAC framework can support future work in long document summarization and coherence evaluation, including improved summarization modeling and posthoc summary correction. 1 Corresponding excerpt from the human-written summary properly contextualizes the new character. Recently, Wu et al. (2021) proposed a strong book summarization model but showed that although generated summaries covered important information from the books, they read like a list of events stapled together without any coherent narrative structure (see Figure 1). We found similar Context: John Fenwick, an aspiring artist, accepts a loan from Mr. Morrison to move to London to pursue his art career. In London, he becomes infatuated with Madame de Pastourelles, a beautiful and intelligent artist.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2205.09641")
get_code_for_paper("2205.09641")
have("2205.09641")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2205.09641.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2205.09641.svg)](https://syntology.ai/paper/2205.09641)

Connect an agent — have() is free.