We lifted 3 functions out of this paper's own repositories and ran 2 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| CogComp/qaeval-experiments | canonical | 1 of 1 |
| copy not recorded | — | 1 of 1 |
| danieldeutsch/qaeval | pwc_unofficial | 0 of 1 |
| Function | Status | Where it lives |
|---|---|---|
| to_list | Ran | this paper's copy was not recorded; identical code first harvested from AI-secure/InfoBERT pointer only · get_code("9df40357afea56cc") |
| load_annotations | Ran | CogComp/qaeval-experiments/experiments/answer-selection/compare_questions_scus.py pointer only (licence: NONE) · get_code("3bdf8cb9b92186db") |
| compute_predictions_logits_with_null | Not yet run | danieldeutsch/qaeval/qaeval/answering/utils.py code served (permissive licence) · get_code("2f456d32384e97b4") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
A desirable property of a reference-based evaluation metric that measures the content quality of a summary is that it should estimate how much information that summary has in common with a reference. Traditional text overlap based metrics such as ROUGE fail to achieve this because they are limited to matching tokens, either lexically or via embeddings. In this work, we propose a metric to evaluate the content quality of a summary using question-answering (QA). QA-based methods directly measure a summary's information overlap with a reference, making them fundamentally different than text overlap metrics. We demonstrate the experimental benefits of QA-based metrics through an analysis of our proposed metric, QAEval. QAEval out-performs current state-of-the-art metrics on most evaluations using benchmark datasets, while being competitive on others due to limitations of state-of-the-art models. Through a careful analysis of each component of QAEval, we identify its performance bottlenecks and estimate that its potential upper-bound performance surpasses all other automatic metrics, approaching that of the gold-standard Pyramid Method.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2010.00490")
get_code_for_paper("2010.00490")
have("2010.00490")
Connect an agent — have() is free.