Luke Zettlemoyer, Sonal Gupta, Armen Aghajanyan, Xilun Chen, Anchit Gupta
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
We propose pre-finetuning, an additional largescale learning stage between language model pre-training and fine-tuning. Pre-finetuning is massively multi-task learning (around 50 datasets, over 4.8 million total labeled examples), and is designed to encourage learning of representations that generalize better to many different tasks. We show that prefinetuning consistently improves performance for pretrained discriminators (e.g. RoBERTa) and generation models (e.g. BART) on a wide range of tasks (sentence prediction, commonsense reasoning, MRC, etc.), while also significantly improving sample efficiency during fine-tuning. We also show that large-scale multi-tasking is crucial; pre-finetuning can hurt performance when few tasks are used up until a critical point (usually above 15) after which performance improves linearly in the number of tasks.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2101.11038")
get_code_for_paper("2101.11038")
have("2101.11038")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2101.11038.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2101.11038)
Connect an agent — have() is free.