We lifted 3 functions out of this paper's own repositories and ran 2 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| assafbk/decimamba | canonical | 2 of 3 |
| Function | Status | Where it lives |
|---|---|---|
| collate_fn_niah | Ran | assafbk/decimamba/finetune_ssm.py code served (permissive licence) · get_code("23ef5bd9b2fce3c4") |
| collate_fn_squad | Ran | assafbk/decimamba/finetune_ssm.py code served (permissive licence) · get_code("dd316fe0068d5597") |
| collate_fn_ppl_test_3 | Not yet run | assafbk/decimamba/finetune_ssm.py code served (permissive licence) · get_code("e30e19c0c12edc6c") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Long-range sequence processing poses a significant challenge for Transformers due to their quadratic complexity in input length. A promising alternative is Mamba, which demonstrates high performance and achieves Transformer-level capabilities while requiring substantially fewer computational resources. In this paper we explore the length-generalization capabilities of Mamba, which we find to be relatively limited. Through a series of visualizations and analyses we identify that the limitations arise from a restricted effective receptive field, dictated by the sequence length used during training. To address this constraint, we introduce DeciMamba, a context-extension method specifically designed for Mamba. This mechanism, built on top of a hidden filtering mechanism embedded within the S6 layer, enables the trained model to extrapolate well even without additional training. Empirical experiments over real-world long-range NLP tasks show that DeciMamba can extrapolate to context lengths that are significantly longer than the ones seen during training, while enjoying faster inference.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2406.14528")
get_code_for_paper("2406.14528")
have("2406.14528")
Connect an agent — have() is free.