Bloomberg, Yufang Hou, Sebastian Gehrmann, João Sedoc, T Darmstadt, Yixin Liu, Khyathi Chandu, Simon Mille, Leonardo Ribeiro, Daniel Deutsch, Saad Mahamood, Miruna Clinciu, and 6 more
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
At the heart of the Pyramid evaluation method for text summarization lie human written summary content units (SCUs). These SCUs are concise sentences that decompose a summary into small facts. Such SCUs can be used to judge the quality of a candidate summary, possibly partially automated via natural language inference (NLI) systems. Interestingly, with the aim to fully automate the Pyramid evaluation, Zhang and Bansal (2021) show that SCUs can be approximated by automatically generated semantic role triplets (STUs). However, several questions currently lack answers, in particular: i) Are there other ways of approximating SCUs that can offer advantages? ii) Under which conditions are SCUs (or their approximations) offering the most value? In this work, we examine two novel strategies to approximate SCUs: generating SCU approximations from AMR meaning representations (SMUs) and from large language models (SGUs), respectively. We find that while STUs and SMUs are competitive, the best approximation quality is achieved by SGUs. We also show through a simple sentence-decomposition baseline (SSUs) that SCUs (and their approximations) offer the most value when ranking short summaries, but may not help as much when ranking systems or longer summaries.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2404.01701")
get_code_for_paper("2404.01701")
have("2404.01701")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2404.01701.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2404.01701)
Connect an agent — have() is free.