Graham Neubig, Pengfei Liu, Manik Bhandari, Pranav Gour, Atabak Ashfaq
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Automated evaluation metrics as a stand-in for manual evaluation are an essential part of the development of text-generation tasks such as text summarization. However, while the field has progressed, our standard metrics have not -for nearly 20 years ROUGE has been the standard evaluation in most summarization papers. In this paper, we make an attempt to re-evaluate the evaluation method for text summarization: assessing the reliability of automatic metrics using top-scoring system outputs, both abstractive and extractive, on recently popular datasets for both systemlevel and summary-level evaluation settings. We find that conclusions about evaluation metrics on older datasets do not necessarily hold on modern datasets and systems. We release a dataset of human judgments that are collected from 25 top-scoring neural summarization systems (14 abstractive and 11 extractive):
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2010.07100")
get_code_for_paper("2010.07100")
have("2010.07100")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2010.07100.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2010.07100)
Connect an agent — have() is free.