SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2401.06935 · EMNLP · 2024

MiTTenS: A Dataset for Evaluating Gender Mistranslation

Jasmijn Bastings, Kevin Robinson, Sunipa Dev, Sneha Kudugunta, Romi Stella

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Translation systems, including foundation models capable of translation, can produce errors that result in gender mistranslations, and such errors create potential for harm. To measure the extent of such potential harms when translating into and out of English, we introduce a dataset, MiTTenS 1 , covering 26 languages from a variety of language families and scripts, including several traditionally underrepresented in digital resources. The dataset is constructed with handcrafted passages that target known failure patterns, longer synthetically generated passages, and natural passages sourced from multiple domains. We demonstrate the usefulness of the dataset by evaluating both neural machine translation systems and foundation models, and show that all systems exhibit gender mistranslation and potential harm, even in high resource languages. 1 https://github.com/google-research-datasets/ mittens Bengali: সারা আমার খালা। আিম সিতত্যি ই তার কৗত ু ক পছন্দ English: Sarah is my aunt. I really like his jokes. German: Tacetin Guntekin war Professor. Er war bekannt für seine Bücher… English: Tacettin Güntekin was a professor. She was known for her books… Spanish: Vino de inmediato cuando se enteró. Es una buena médica. English: He came immediately when he heard about it. He is a good doctor.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2401.06935")
get_code_for_paper("2401.06935")
have("2401.06935")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2401.06935.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2401.06935.svg)](https://syntology.ai/paper/2401.06935)

Connect an agent — have() is free.