SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2104.06387 · ACL · 2021

EXPLAINABOARD: An Explainable Leaderboard for NLP

Graham Neubig, Yang Xiao, Weizhe Yuan, Pengfei Liu, Shuaichen Chang, Yixin Liu, Jinlan Fu, Zihuiwen Ye, Junqi Dai

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

With the rapid development of NLP research, leaderboards have emerged as one tool to track the performance of various systems on various NLP tasks. They are effective in this goal to some extent, but generally present a rather simplistic one-dimensional view of the submitted systems, communicated only through holistic accuracy numbers. In this paper, we present a new conceptualization and implementation of NLP evaluation: the EXPLAIN-ABOARD, which in addition to inheriting the functionality of the standard leaderboard, also allows researchers to (i) diagnose strengths and weaknesses of a single system (e.g. what is the best-performing system bad at?) (ii) interpret relationships between multiple systems. (e.g. where does system A outperform system B? What if we combine systems A, B and C?) and (iii) examine prediction results closely (e.g. what are common errors made by multiple systems or in what contexts do particular errors occur?). So far, EXPLAINABOARD covers more than 400 systems, 50 datasets, 40 languages, and 12 tasks. 1 We not only released an online platform at the website 2 but also make our evaluation tool an API with MIT Licence at Github 3 and PyPi 4 that allows users to conveniently assess their models offline. We additionally release all output files from systems that we have run or collected to motivate "output-driven" research in the future.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2104.06387")
get_code_for_paper("2104.06387")
have("2104.06387")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2104.06387.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2104.06387.svg)](https://syntology.ai/paper/2104.06387)

Connect an agent — have() is free.