Furu Wei, Ting Liu, Xia Song, Li Dong, Bo Zheng, Shaohan Huang, Saksham Singhal, Wanxiang Che
We lifted 2 functions out of this paper's own repositories and ran 1 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| bozheng-hit/vocapxlm | — | 1 of 1 |
| hit-scir/chinese-mixtral-8x7b | — | 0 of 1 |
| Function | Status | Where it lives |
|---|---|---|
| merge_vocab | Ran | bozheng-hit/vocapxlm/train_vocap.py pointer only (licence: NONE) · get_code("de70d8e5b33f3e0c") |
| init_embeddings_average | Not yet run | hit-scir/chinese-mixtral-8x7b/models/init_embeddings.py code served (permissive licence) · get_code("e4748d46187b73d7") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Compared to monolingual models, crosslingual models usually require a more expressive vocabulary to represent all languages adequately. We find that many languages are under-represented in recent cross-lingual language models due to the limited vocabulary capacity. To this end, we propose an algorithm VOCAP to determine the desired vocabulary capacity of each language. However, increasing the vocabulary size significantly slows down the pre-training speed. In order to address the issues, we propose k-NN-based target sampling to accelerate the expensive softmax. Our experiments show that the multilingual vocabulary learned with VOCAP benefits cross-lingual language model pre-training. Moreover, k-NN-based target sampling mitigates the side-effects of increasing the vocabulary size while achieving comparable performance and faster pre-training speed. The code and the pretrained multilingual vocabularies are available at https://github. com/bozheng-hit/VoCapXLM.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2109.07306")
get_code_for_paper("2109.07306")
have("2109.07306")
Connect an agent — have() is free.