SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2105.02855 · ACL Findings · 2021

Adapting Monolingual Models: Data can be Scarce when Language Similarity is High

Malvina Nissim, Wietse De Vries, Martijn Bartelds, Martijn Wieling

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

For many (minority) languages, the resources needed to train large models are not available. We investigate the performance of zero-shot transfer learning with as little data as possible, and the influence of language similarity in this process. We retrain the lexical layers of four BERT-based models using data from two low-resource target language varieties, while the Transformer layers are independently finetuned on a POS-tagging task in the model's source language. By combining the new lexical layers and fine-tuned Transformer layers, we achieve high task performance for both target languages. With high language similarity, 10MB of data appears sufficient to achieve substantial monolingual transfer performance. Monolingual BERT-based models generally achieve higher downstream task performance after retraining the lexical layer than multilingual BERT, even when the target language is included in the multilingual model.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2105.02855")
get_code_for_paper("2105.02855")
have("2105.02855")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2105.02855.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2105.02855.svg)](https://syntology.ai/paper/2105.02855)

Connect an agent — have() is free.