Zhen Qin, Dongxu Li, Yiran Zhong, Lingpeng Kong, Han, Nick Barnes, Weixuan Sun
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Linear transformers aim to reduce the quadratic space-time complexity of vanilla transformers. However, they usually suffer from degraded performances on various tasks and corpora. In this paper, we examine existing kernel-based linear transformers and identify two key issues that lead to such performance gaps: 1) unbounded gradients in the attention computation adversely impact the convergence of linear transformer models; 2) attention dilution which trivially distributes attention scores over long sequences while neglecting neighbouring structures. To address these issues, we first identify that the scaling of attention matrices is the devil in unbounded gradients, which turns out unnecessary in linear attention as we show theoretically and empirically. To this end, we propose a new linear attention that replaces the scaling operation with a normalization to stabilize gradients. For the issue of attention dilution, we leverage a diagonal attention to confine attention to only neighbouring tokens in early layers. Benefiting from the stable gradients and improved attention, our new linear transformer model, TRANSNORMER, demonstrates superior performance on text classification and language modeling tasks, as well as on the challenging Long-Range Arena benchmark, surpassing vanilla transformer and existing linear variants by a clear margin while being significantly more space-time efficient. The code is available at TRANSNORMER.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2210.10340")
get_code_for_paper("2210.10340")
have("2210.10340")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2210.10340.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2210.10340)
Connect an agent — have() is free.