SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2402.13249 · NAACL · 2024

TOFUEVAL: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue Summarization

Hang Su, Yi Zhang, Hwanjun Song, Kathleen Mckeown, Liyan Tang, Saab Mansour, Siffi Singh, Igor Shalyminov, Song Feng, Amy Wing-Mei Wong, Jon Burnsky, Jake Vincent, and 2 more

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Single document news summarization has seen substantial progress on faithfulness in recent years, driven by research on the evaluation of factual consistency, or hallucinations. We ask whether these advances carry over to other text summarization domains. We propose a new evaluation benchmark on topic-focused dialogue summarization, generated by LLMs of varying sizes. We provide binary sentencelevel human annotations of the factual consistency of these summaries along with detailed explanations of factually inconsistent sentences. Our analysis shows that existing LLMs hallucinate significant amounts of factual errors in the dialogue domain, regardless of the model's size. On the other hand, when LLMs, including GPT-4, serve as binary factual evaluators, they perform poorly and can be outperformed by prevailing state-of-the-art specialized factuality evaluation metrics. Finally, we conducted an analysis of hallucination types with a curated error taxonomy. We find that there are diverse errors and error distributions in modelgenerated summaries and that non-LLM based metrics can capture all error types better than LLM-based evaluators. 1

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2402.13249")
get_code_for_paper("2402.13249")
have("2402.13249")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2402.13249.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2402.13249.svg)](https://syntology.ai/paper/2402.13249)

Connect an agent — have() is free.