Tao Lei
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Large language models have become increasingly difficult to train because of the growing computation time and cost. In this work, we present SRU++, a highly-efficient architecture that combines fast recurrence and attention for sequence modeling. SRU++ exhibits strong modeling capacity and training efficiency. On standard language modeling tasks such as ENWIK8, WIKI-103 and BIL-LION WORD datasets, our model obtains better bits-per-character and perplexity while using 3x-10x less training cost compared to topperforming Transformer models. For instance, our model achieves a state-of-the-art result on the ENWIK8 dataset using 1.6 days of training on an 8-GPU machine. We further demonstrate that SRU++ requires minimal attention for near state-of-the-art performance. Our results suggest jointly leveraging fast recurrence with little attention as a promising direction for accelerating model training and inference. 1
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2102.12459")
get_code_for_paper("2102.12459")
have("2102.12459")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2102.12459.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2102.12459)
Connect an agent — have() is free.