Minlie Huang, Pei Ke, Siyang Liu, Xiaoyan Zhu, Yinhe Zheng, Sahand Sabour
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. If it links a repository, it is listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Distinct-n score (Li et al., 2016) is a widely used automatic metric for evaluating diversity in language generation tasks. However, we observed that the original approach for calculating distinct scores has evident biases that tend to assign higher penalties to longer sequences. We refine the calculation of distinct scores by scaling the number of distinct tokens based on their expectations. We provide both empirical and theoretical evidence to show that our method effectively removes the biases existing in the original distinct score. Our experiments show that our proposed metric, Expectation-Adjusted Distinct (EAD), correlates better with human judgment in evaluating response diversity. To foster future research, we provide an example implementation at https://github.com/lsy641/ Expectation-Adjusted-Distinct.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2202.13587")
get_code_for_paper("2202.13587")
have("2202.13587")
Connect an agent — have() is free.