SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2301.10472 · EMNLP · 2023

XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models

Marjan Ghazvininejad, Rui Hou, Naman Goyal, Luke Zettlemoyer, Davis Liang, Yuning Mao, Madian Khabsa, Hila Gonen

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Large multilingual language models typically rely on a single vocabulary shared across 100+ languages. As these models have increased in parameter count and depth, vocabulary size has remained largely unchanged. This vocabulary bottleneck limits the representational capabilities of multilingual models like XLM-R. In this paper, we introduce a new approach for scaling to very large multilingual vocabularies by de-emphasizing token sharing between languages with little lexical overlap and assigning vocabulary capacity to achieve sufficient coverage for each individual language. Tokenizations using our vocabulary are typically more semantically meaningful and shorter compared to XLM-R. Leveraging this improved vocabulary, we train XLM-V, a multilingual language model with a one million token vocabulary. XLM-V outperforms XLM-R on every task we tested on ranging from natural language inference (XNLI), question answering (MLQA, XQuAD, TyDiQA), to named entity recognition (WikiAnn). XLM-V is particularly effective on low-resource language tasks and outperforms XLM-R by 11.2% and 5.8% absolute on MasakhaNER and Americas NLI, respectively.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2301.10472")
get_code_for_paper("2301.10472")
have("2301.10472")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2301.10472.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2301.10472.svg)](https://syntology.ai/paper/2301.10472)

Connect an agent — have() is free.