SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2301.09008 · 2023

Poor Man's Quality Estimation: Predicting Reference-Based MT Metrics Without the Reference

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Machine translation quality estimation (QE) predicts human judgements of a translation hypothesis without seeing the reference. State-of-the-art QE systems based on pretrained language models have been achieving remarkable correlations with human judgements yet they are computationally heavy and require human annotations, which are slow and expensive to create. To address these limitations, we define the problem of metric estimation (ME) where one predicts the automated metric scores also without the reference. We show that even without access to the reference, our model can estimate automated metrics ($ρ$=60% for BLEU, $ρ$=51% for other metrics) at the sentence-level. Because automated metrics correlate with human judgements, we can leverage the ME task for pre-training a QE model. For the QE task, we find that pre-training on TER is better ($ρ$=23%) than training for scratch ($ρ$=20%).

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2301.09008")
get_code_for_paper("2301.09008")
have("2301.09008")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2301.09008.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2301.09008.svg)](https://syntology.ai/paper/2301.09008)

Connect an agent — have() is free.