Arber Zela, Julien Siems, Riccardo Grazzi, Massimiliano Pontil, Timur Carstensen, Frank Hutter △♣♢
We lifted 2 functions out of this paper's own repositories and ran 2 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| automl/deltaproduct | canonical | 2 of 2 |
| Function | Status | Where it lives |
|---|---|---|
| elu_p1 | Ran | automl/deltaproduct/flash-linear-attention/fla/layers/gated_deltaproduct.py code served (permissive licence) · get_code("ee8f82afd691729b") |
| sum_norm | Ran | automl/deltaproduct/flash-linear-attention/fla/layers/gated_deltaproduct.py code served (permissive licence) · get_code("e544fdb6373b9f42") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Linear Recurrent Neural Networks (linear RNNs) have emerged as competitive alternatives to Transformers for sequence modeling, offering efficient training and linear-time inference. However, existing architectures face a fundamental tradeoff between expressivity and efficiency, dictated by the structure of their statetransition matrices. Diagonal matrices, used in models such as Mamba, GLA, or mLSTM, yield fast runtime but have limited expressivity. To address this, recent architectures such as DeltaNet and RWKV-7 adopted a diagonal plus rank-1 structure, which allows simultaneous token and channel mixing, improving associative recall and, as recently shown, state-tracking when allowing state-transition matrices to have negative eigenvalues. Building on the interpretation of DeltaNet's recurrence as performing one step of online gradient descent per token on an associative recall loss, we introduce DeltaProduct, which instead takes multiple (n h ) steps per token. This naturally leads to diagonal plus rank-n h state-transition matrices, formed as products of n h generalized Householder transformations, providing a tunable mechanism to balance expressivity and efficiency. We provide a detailed theoretical characterization of the state-tracking capability of DeltaProduct in finite precision, showing how it improves by increasing n h . Our extensive experiments demonstrate that DeltaProduct outperforms DeltaNet in both statetracking and language modeling, while also showing significantly improved length extrapolation capabilities.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2502.10297")
get_code_for_paper("2502.10297")
have("2502.10297")
Connect an agent — have() is free.