SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2112.14569 · 2021

Fine-Tuning Transformers: Vocabulary Transfer

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Transformers are responsible for the vast majority of recent advances in natural language processing. The majority of practical natural language processing applications of these models are typically enabled through transfer learning. This paper studies if corpus-specific tokenization used for fine-tuning improves the resulting performance of the model. Through a series of experiments, we demonstrate that such tokenization combined with the initialization and fine-tuning strategy for the vocabulary tokens speeds up the transfer and boosts the performance of the fine-tuned model. We call this aspect of transfer facilitation vocabulary transfer.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2112.14569")
get_code_for_paper("2112.14569")
have("2112.14569")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2112.14569.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2112.14569.svg)](https://syntology.ai/paper/2112.14569)

Connect an agent — have() is free.