Yang Gao, Wei Zhao, Steffen Eger, Maxime Peyrard, Robert West, Goran Glavaš
We lifted 3 functions out of this paper's own repositories and ran 3 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| AIPHES/ACL20-Reference-Free-MT-Evaluation | canonical | 2 of 2 |
| copy not recorded | — | 1 of 1 |
| Function | Status | Where it lives |
|---|---|---|
| bert_encode | Ran | AIPHES/ACL20-Reference-Free-MT-Evaluation/score_utils.py code served (permissive licence) · get_code("78010dc51e6ba072") |
| collate_idf | Ran | AIPHES/ACL20-Reference-Free-MT-Evaluation/score_utils.py code served (permissive licence) · get_code("d9cba0455da2e616") |
| padding | Ran | this paper's copy was not recorded; identical code first harvested from yuhui-zh15/nlg_metrics pointer only · get_code("13ef8eeae8570519") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Evaluation of cross-lingual encoders is usually performed either via zero-shot cross-lingual transfer in supervised downstream tasks or via unsupervised cross-lingual textual similarity. In this paper, we concern ourselves with reference-free machine translation (MT) evaluation where we directly compare source texts to (sometimes low-quality) system translations, which represents a natural adversarial setup for multilingual encoders. Referencefree evaluation holds the promise of web-scale comparison of MT systems. We systematically investigate a range of metrics based on state-of-the-art cross-lingual semantic representations obtained with pretrained M-BERT and LASER. We find that they perform poorly as semantic encoders for reference-free MT evaluation and identify their two key limitations, namely, (a) a semantic mismatch between representations of mutual translations and, more prominently, (b) the inability to punish "translationese", i.e., low-quality literal translations. We propose two partial remedies: (1) post-hoc re-alignment of the vector spaces and (2) coupling of semantic-similarity based metrics with target-side language modeling. In segment-level MT evaluation, our best metric surpasses reference-based BLEU by 5.7 correlation points. We make our MT evaluation code available. 1
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2005.01196")
get_code_for_paper("2005.01196")
have("2005.01196")
Connect an agent — have() is free.