SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2004.06100 · ACL · 2020

Pretrained Transformers Improve Out-of-Distribution Robustness

U Berkeley, Eric Wallace, Dan Hendrycks, Adam Dziedzic, Dawn Song, Shanghai Jiao, Xiaoyuan Liu, Rishabh Krishnan

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Although pretrained Transformers such as BERT achieve high accuracy on indistribution examples, do they generalize to new distributions? We systematically measure out-of-distribution (OOD) generalization for seven NLP datasets by constructing a new robustness benchmark with realistic distribution shifts. We measure the generalization of previous models including bag-of-words models, ConvNets, and LSTMs, and we show that pretrained Transformers' performance declines are substantially smaller. Pretrained transformers are also more effective at detecting anomalous or OOD examples, while many previous models are frequently worse than chance. We examine which factors affect robustness, finding that larger models are not necessarily more robust, distillation can be harmful, and more diverse pretraining data can enhance robustness. Finally, we show where future work can improve OOD robustness.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2004.06100")
get_code_for_paper("2004.06100")
have("2004.06100")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2004.06100.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2004.06100.svg)](https://syntology.ai/paper/2004.06100)

Connect an agent — have() is free.