Ali Razavi, Jack Rae
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Deep attention models have advanced the modelling of sequential data across many domains. For language modelling in particular, the Transformer-XL -a Transformer augmented with a long-range memory of past activations -has been shown to be state-ofthe-art across a variety of well-studied benchmarks. The Transformer-XL incorporates a long-range memory at every layer of the network, which renders its state to be thousands of times larger than RNN predecessors. However it is unclear whether this is necessary. We perform a set of interventions to show that comparable performance can be obtained with 6X fewer long range memories and better performance can be obtained by limiting the range of attention in lower layers of the network.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2007.03356")
get_code_for_paper("2007.03356")
have("2007.03356")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2007.03356.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2007.03356)
Connect an agent — have() is free.