Yixin Nie, Kyle Richardson, Hai Hu, He Zhou, Zuoyu Tian, Yiwen Zhang, Yina Ma, Yanting Li
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Multilingual transformers (XLM, mT5) have been shown to have remarkable transfer skills in zero-shot settings. Most transfer studies, however, rely on automatically translated resources (XNLI, XQuAD), making it hard to discern the particular linguistic knowledge that is being transferred, and the role of expert annotated monolingual datasets when developing task-specific models. We investigate the cross-lingual transfer abilities of XLM-R for Chinese and English natural language inference (NLI), with a focus on the recent largescale Chinese dataset OCNLI. To better understand linguistic transfer, we created 4 categories of challenge and adversarial tasks (totaling 17 new datasets 1 ) for Chinese that build on several well-known resources for English (e.g., HANS, NLI stress-tests). We find that cross-lingual models trained on English NLI do transfer well across our Chinese tasks (e.g., in 3/4 of our challenge categories, they perform as well/better than the best monolingual models, even on 3/5 uniquely Chinese linguistic phenomena such as idioms, pro drop). These results, however, come with important caveats: cross-lingual models often perform best when trained on a mixture of English and high-quality monolingual NLI data (OCNLI), and are often hindered by automatically translated resources (XNLI-zh). For many phenomena, all models continue to struggle, highlighting the need for our new diagnostics to help benchmark Chinese and cross-lingual models. category n Chinese HANS Lexical overlap 1,428 Subsequence 513 stress tests Distraction: 2 categories 8,000 Antonym 3,000 Synonym 2,000 Spelling 11,676 Numerical reasoning 8,613 diagnostics CLUE (Xu et al., 2020) 514 CLUE expansion (ours) 796 World knowledge (ours) 38 Classifier (ours) 139 Chengyu/idioms (ours) 251 Pro-drop (ours) 198 Non-core arguments (ours) 186 semantic probing Negation 1
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2106.03983")
get_code_for_paper("2106.03983")
have("2106.03983")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2106.03983.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2106.03983)
Connect an agent — have() is free.