SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2004.14564 · EMNLP · 2020

Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing

Matt Post, Brian Thompson

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

We frame the task of machine translation evaluation as one of scoring machine translation output with a sequence-to-sequence paraphraser, conditioned on a human reference. We propose training the paraphraser as a multilingual NMT system, treating paraphrasing as a zero-shot translation task (e.g., Czech to Czech). This results in the paraphraser's output mode being centered around a copy of the input sequence, which represents the best case scenario where the MT system output matches a human reference. Our method is simple and intuitive, and does not require human judgements for training. Our single model (trained in 39 languages) outperforms or statistically ties with all prior metrics on the WMT 2019 segment-level shared metrics task in all languages (excluding Gujarati where the model had no training data). We also explore using our model for the task of quality estimation as a metric-conditioning on the source instead of the reference-and find that it significantly outperforms every submission to the WMT 2019 shared task on quality estimation in every language pair. Word-level paraphraser log probabilities H(out|in) sBLEU LASER Jason went to school at the University of Madrid . <EOS> -0.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2004.14564")
get_code_for_paper("2004.14564")
have("2004.14564")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2004.14564.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2004.14564.svg)](https://syntology.ai/paper/2004.14564)

Connect an agent — have() is free.