SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2206.11249 · EMNLP · 2022

GEMv2: Multilingual NLG Benchmarking in a Single Line of Code

Paul Liang, James Cross, Alexandra Luccioni, Chandra Bhagavatula, Aman Madaan, Yi Tay, Donald Metzler, Zhe Zhao, Alexey Gritsenko, Mostafa Dehghani, Qi Zhu, Dragomir Radev, and 133 more

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Evaluations in machine learning rarely use the latest metrics, datasets, or human evaluation in favor of remaining compatible with prior work. The compatibility, often facilitated through leaderboards, thus leads to outdated but standardized evaluation practices. We pose that the standardization is taking place in the wrong spot. Evaluation infrastructure should enable researchers to use the latest methods and what should be standardized instead is how to incorporate these new evaluation advances. We introduce GEMv2, the new version of the Generation, Evaluation, and Metrics Benchmark which uses a modular infrastructure for dataset, model, and metric developers to benefit from each other's work. GEMv2 supports 40 documented datasets in 51 languages, ongoing online evaluation for all datasets, and our interactive tools make it easier to add new datasets to the living benchmark.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2206.11249")
get_code_for_paper("2206.11249")
have("2206.11249")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2206.11249.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2206.11249.svg)](https://syntology.ai/paper/2206.11249)

Connect an agent — have() is free.