SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2211.01355 · EMNLP · 2022

MT-GenEval: A Counterfactual and Contextual Dataset for Evaluating Gender Accuracy in Machine Translation

Benjamin Hsu, Anna Currey, Xing Niu, Maria Nȃdejde, Georgiana Dinu, Raghavendra Pappagari, Mia Mayer, Stanislas Lauly

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

As generic machine translation (MT) quality has improved, the need for targeted benchmarks that explore fine-grained aspects of quality has increased (Freitag et al., 2021;Isabelle et al., 2017). In particular, gender accuracy in translation (Choubey et al., 2021; Saunders and Byrne, 2020) can have implications in terms of output fluency, translation accuracy, and ethics. In this paper, we introduce MT-GenEval, a benchmark for evaluating gender accuracy in translation from English into eight widely-spoken languages. MT-GenEval complements existing benchmarks by providing realistic, gender-balanced, counterfactual data in eight language pairs where the gender of individuals is unambiguous in the input segment, including multi-sentence segments requiring inter-sentential gender agreement. Our data and code is publicly available under a CC BY SA 3.0 license. 1

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2211.01355")
get_code_for_paper("2211.01355")
have("2211.01355")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2211.01355.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2211.01355.svg)](https://syntology.ai/paper/2211.01355)

Connect an agent — have() is free.