SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2010.01150 · EMNLP Findings · 2020

Cost-effective Selection of Pretraining Data: A Case Study of Pretraining BERT on Social Media

Xiang Dai, Sarvnaz Karimi, Ben Hachey, Cecile Paris

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. We have no repository linked to it either. That is a statement about our graph, not about the paper.

Abstract

Recent studies on domain-specific BERT models show that effectiveness on downstream tasks can be improved when models are pretrained on in-domain data. Often, the pretraining data used in these models are selected based on their subject matter, e.g., biology or computer science. Given the range of applications using social media text, and its unique language variety, we pretrain two models on tweets and forum text respectively, and empirically demonstrate the effectiveness of these two resources. In addition, we investigate how similarity measures can be used to nominate in-domain pretraining data. We publicly release our pretrained models at https://bit.ly/35RpTf0. Research question * The selection of in-domain data usually resorts to human intuition. * Can we use simple similarity measures to nominate in-domain data? BERT Target Labeled Data In-domain data Standard approach * Pretrain BERT from scratch on generic data * Fine-tune BERT on target labeled data Domain-specific BERT * Pretrain BERT from scratch on generic data * Continue pretraining BERT on domain-specific corpus (in-domain data) * Fine-tune BERT on target labeled data

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2010.01150")
get_code_for_paper("2010.01150")
have("2010.01150")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2010.01150.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2010.01150.svg)](https://syntology.ai/paper/2010.01150)

Connect an agent — have() is free.