SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2210.13832 · EMNLP · 2022

FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluation

Chen Zhang, Haizhou Li, Luis Fernando D'haro, Qiquan Zhang, Thomas Friedrichs

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Recent model-based reference-free metrics for open-domain dialogue evaluation exhibit promising correlations with human judgment 1 . However, they either perform turn-level evaluation or look at a single dialogue quality dimension. One would expect a good evaluation metric to assess multiple quality dimensions at the dialogue level. To this end, we are motivated to propose a multi-dimensional dialogue-level metric, which consists of three sub-metrics with each targeting a specific dimension. The submetrics are trained with novel self-supervised objectives and exhibit strong correlations with human judgment for their respective dimensions. Moreover, we explore two approaches to combine the sub-metrics: metric ensemble and multitask learning. Both approaches yield a holistic metric that significantly outperforms individual sub-metrics. Compared to the existing state-of-the-art metric, the combined metrics achieve around 16% relative improvement on average across three high-quality dialoguelevel evaluation benchmarks.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2210.13832")
get_code_for_paper("2210.13832")
have("2210.13832")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2210.13832.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2210.13832.svg)](https://syntology.ai/paper/2210.13832)

Connect an agent — have() is free.