SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2102.11972 · EMNLP · 2021

Do Transformer Modifications Transfer Across Implementations and Applications?

Colin Raffel, William Fedus, Adam Roberts, Yanqi Zhou, Zhenzhong Lan, Wei Li, Noam Shazeer, Sharan Narang, Noah Fiedel, Hyung Won, Thibault Fevry, Michael Matena, and 3 more

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

The research community has proposed copious modifications to the Transformer architecture since it was introduced over three years ago, relatively few of which have seen widespread adoption. In this paper, we comprehensively evaluate many of these modifications in a shared experimental setting that covers most of the common uses of the Transformer in natural language processing. Surprisingly, we find that most modifications do not meaningfully improve performance. Furthermore, most of the Transformer variants we found beneficial were either developed in the same codebase that we used or are relatively minor changes. We conjecture that performance improvements may strongly depend on implementation details and correspondingly make some recommendations for improving the generality of experimental results.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2102.11972")
get_code_for_paper("2102.11972")
have("2102.11972")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2102.11972.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2102.11972.svg)](https://syntology.ai/paper/2102.11972)

Connect an agent — have() is free.