SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2104.07412 · EMNLP · 2021

XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation

Graham Neubig, Junjie Hu, Sebastian Ruder, Aditya Siddhant, Orhan Firat, Melvin Johnson, Pengfei Liu, Noah Constant, Jinlan Fu, Dan Garrette, Jan Botha

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Machine learning has brought striking advances in multilingual natural language processing capabilities over the past year. For example, the latest techniques have improved the state-of-the-art performance on the XTREME multilingual benchmark by more than 13 points. While a sizeable gap to humanlevel performance remains, improvements have been easier to achieve in some tasks than in others. This paper analyzes the current state of cross-lingual transfer learning and summarizes some lessons learned. In order to catalyze meaningful progress, we extend XTREME to XTREME-R, which consists of an improved set of ten natural language understanding tasks, including challenging language-agnostic retrieval tasks, and covers 50 typologically diverse languages. In addition, we provide a massively multilingual diagnostic suite (MULTICHECKLIST) and finegrained multi-dataset evaluation capabilities through an interactive public leaderboard to gain a better understanding of such models.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2104.07412")
get_code_for_paper("2104.07412")
have("2104.07412")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2104.07412.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2104.07412.svg)](https://syntology.ai/paper/2104.07412)

Connect an agent — have() is free.