Tian Xie, Ran Tao, Zican Dong, Yucheng Ding, Yiwen Hu, Wayne Xin Zhao, Bryan Dai, Zian Liu, Zi-Ang Liu, Yi-Wen Hu, Yu-Cheng Ding
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Recently, linear attention layers have been increasingly adopted to replace softmax attention at scale for long-context modeling. However, existing context extension approaches typically apply continued pretraining directly without modifying these layers, overlooking the spectral properties of linear attention state dynamics. In this work, we study long-context extension of Gated DeltaNet (GDN) from a spectral perspective of transition matrix and identify two essential factors governing long-range information retrieval: (1) a sufficiently broad slow spectral band aligned with the target dependency length, and (2) the preservation of fast-decaying modes for state clearing and context switching. Based on this observation, we propose SpectralShift, a spectral reparameterization approach for long-context continual pretraining of GDNs. Specifically, SpectralShift reparameterizes the alpha projections initialization to reshape the decay spectrum by enhancing slow propagation capacity, and further introduces a learning-rate scaling for alpha projections to facilitate long-context training. Experiments show that SpectralShift consistently improves long-context capabilities over training, providing an effective and efficient solution for extending context windows of linear attention models. The code has been open-sourced at https://github.com/RUCAIBox/GDN-SpectralShift.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2609.14320")
get_code_for_paper("2609.14320")
have("2609.14320")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2609.14320.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2609.14320)
Connect an agent — have() is free.