Kevin Knight, Ajay Nagesh, Boliang Zhang
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. We have no repository linked to it either. That is a statement about our graph, not about the paper.
Web-crawled data provides a good source of parallel corpora for training machine translation models. It is automatically obtained, but extremely noisy, and recent work shows that neural machine translation systems are more sensitive to noise than traditional statistical machine translation methods. In this paper, we propose a novel approach to filter out noisy sentence pairs from web-crawled corpora via pre-trained language models. We measure sentence parallelism by leveraging the multilingual capability of BERT and use the Generative Pre-training (GPT) language model as a domain filter to balance data domains. We evaluate the proposed method on the WMT 2018 Parallel Corpus Filtering shared task, and on our own web-crawled Japanese-Chinese parallel corpus. Our method significantly outperforms baselines and achieves a new stateof-the-art. In an unsupervised setting, our method achieves comparable performance to the top-1 supervised method. We also evaluate on a web-crawled Japanese-Chinese parallel corpus that we make publicly available.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2005.06166")
get_code_for_paper("2005.06166")
have("2005.06166")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2005.06166.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2005.06166)
Connect an agent — have() is free.