We lifted 13 functions out of this paper's own repositories and ran 10 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| recursal/radlads-paper | canonical | 10 of 13 |
| Function | Status | Where it lives |
|---|---|---|
| repeat_kv | Ran | recursal/radlads-paper/rwkv6attn.py code served (permissive licence) · get_code("3c76e52815c5401d") |
| apply_rotary_pos_emb | Ran | recursal/radlads-paper/rwkv6qwen2/modeling_rwkv6qwen2.py code served (permissive licence) · get_code("bac65c3dafaec040") |
| apply_rotary_pos_emb | Ran | recursal/radlads-paper/models/qwen2.py code served (permissive licence) · get_code("67e215883a9a54b6") |
| convert_dict_to_config | Ran | recursal/radlads-paper/configs.py code served (permissive licence) · get_code("e8df29c14266e90d") |
| generate_rotary_embedding | Ran | recursal/radlads-paper/models/qwen2.py code served (permissive licence) · get_code("fe9b629d83cc3089") |
| is_prime | Ran | recursal/radlads-paper/make_data_hf.py code served (permissive licence) · get_code("b5bc417f90245613") |
| literal_eval | Ran | recursal/radlads-paper/configs.py code served (permissive licence) · get_code("82ae923d20eb7fcb") |
| load_checkpoint | Ran | recursal/radlads-paper/stack_layers.py code served (permissive licence) · get_code("91befd9a752c49a4") |
| merge_config | Ran | recursal/radlads-paper/configs.py code served (permissive licence) · get_code("9c3d0d0b3af30bb8") |
| rotate_half | Ran | recursal/radlads-paper/models/qwen2.py code served (permissive licence) · get_code("b99eea6376d1e212") |
| data_file_path | Not yet run | recursal/radlads-paper/make_data_hf.py code served (permissive licence) · get_code("bdf28f9492a770dc") |
| index_file_path | Not yet run | recursal/radlads-paper/make_data_hf.py code served (permissive licence) · get_code("8b1a74a6db32c163") |
| load_and_patch_model_with_attention_replacement | Not yet run | recursal/radlads-paper/rwkv6attn.py code served (permissive licence) · get_code("0b8395d5f5512df0") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
We present Rapid Attention Distillation to Linear Attention Decoders at Scale (RADLADS), a protocol for rapidly converting softmax attention transformers into linear attention decoder models, along with two new RWKV-variant architectures, and models converted from popular Qwen2.5 open source models in 7B, 32B, and 72B sizes. Our conversion process requires only 350-700M tokens, less than 0.005% of the token count used to train the original teacher models. Converting to our 72B linear attention model costs less than \$2,000 USD at today's prices, yet quality at inference remains close to the original transformer. These models achieve state-of-the-art downstream performance across a set of standard benchmarks for linear attention models of their size. We release all our models on HuggingFace under the Apache 2.0 license, with the exception of our 72B models which are also governed by the Qwen License Agreement. Models at https://huggingface.co/collections/recursal/radlads-6818ee69e99e729ba8a87102 Training Code at https://github.com/recursal/RADLADS-paper
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2505.03005")
get_code_for_paper("2505.03005")
have("2505.03005")
Connect an agent — have() is free.