Yu Lu, Jianlin Su, Bo Wen, Shengfeng Pan, Ahmed Murtadha, Yunfeng Liu
We lifted 16 functions out of this paper's own repositories and ran 14 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| varungumma/fairseq | — | 3 of 3 |
| lucidrains/rotary-embedding-torch | — | 3 of 3 |
| AryaAftab/rotary-embedding-tensorflow | — | 2 of 2 |
| willGuimont/transformers | — | 1 of 2 |
| singaln/Roformer_Simlarity | — | 1 of 1 |
| labmlai/annotated_deep_learning_paper_implementations | — | 1 of 1 |
| airi-institute/gena_lm | — | 1 of 1 |
| baichuan-inc/baichuan-7b | — | 1 of 1 |
| lucidrains/reformer-pytorch | — | 1 of 1 |
| JunnYu/RoFormer_pytorch | — | 0 of 1 |
| Function | Status | Where it lives |
|---|---|---|
| MultiHeadsAttention | Ran | singaln/Roformer_Simlarity/Rotransformer.py code served (permissive licence) · get_code("ec6ef1c866e39c6f") |
| RoPE | Ran | labmlai/annotated_deep_learning_paper_implementations/labml_nn/neox/model.py code served (permissive licence) · get_code("8f063cad604c5b86") |
| RotaryEmbedding | Ran | AryaAftab/rotary-embedding-tensorflow/rotary_embedding_tensorflow/rotary_embedding_tensorflow.py code served (permissive licence) · get_code("0fe09c685e79fefc") |
| RotaryEmbedding | Ran | varungumma/fairseq/fairseq/modules/rotary_embedding.py code served (permissive licence) · get_code("dbb6d6a0d27a25f4") |
| RotaryEmbedding | Ran | airi-institute/gena_lm/src/gena_lm/modeling_bert.py code served (permissive licence) · get_code("9a156a004cf5913c") |
| RotaryEmbedding | Ran | lucidrains/rotary-embedding-torch/rotary_embedding_torch/rotary_embedding_torch.py code served (permissive licence) · get_code("0a4292dfdf0c449d") |
| RotaryEmbedding | Ran | baichuan-inc/baichuan-7b/models/modeling_baichuan.py code served (permissive licence) · get_code("d05c5eb6cdfb5b97") |
| apply_rotary_emb | Ran | varungumma/fairseq/fairseq/modules/rotary_embedding.py code served (permissive licence) · get_code("60f6b43d9fa0bb55") |
| apply_rotary_emb | Ran | lucidrains/rotary-embedding-torch/rotary_embedding_torch/rotary_embedding_torch.py code served (permissive licence) · get_code("0a495c8bcfb0380b") |
| apply_rotary_pos_emb | Ran | lucidrains/reformer-pytorch/reformer_pytorch/reformer_pytorch.py code served (permissive licence) · get_code("3c0f5df83f530eba") |
| relative_positional_encoding | Ran | willGuimont/transformers/nnet/positional_encoding/relative_positional_encoding.py code served (permissive licence) · get_code("e37b416bd6989338") |
| repeat | Ran | AryaAftab/rotary-embedding-tensorflow/rotary_embedding_tensorflow/rotary_embedding_tensorflow.py code served (permissive licence) · get_code("4c8c38feed7983ab") |
| rotate_half | Ran | varungumma/fairseq/fairseq/modules/rotary_embedding.py code served (permissive licence) · get_code("58823d9435a8751b") |
| slice_at_dim | Ran | lucidrains/rotary-embedding-torch/rotary_embedding_torch/rotary_embedding_torch.py code served (permissive licence) · get_code("50c38f60f1badef7") |
| RelativePositionalEncoding | Not yet run | willGuimont/transformers/nnet/positional_encoding/relative_positional_encoding.py code served (permissive licence) · get_code("c4c1375689d7d3dc") |
| RoFormerSelfAttention | Not yet run | JunnYu/RoFormer_pytorch/src/roformer/modeling_roformer.py code served (permissive licence) · get_code("5e8d748cacb045bd") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Position encoding recently has shown effective in the transformer architecture. It enables valuable supervision for dependency modeling between elements at different positions of the sequence. In this paper, we first investigate various methods to integrate positional information into the learning process of transformer-based language models. Then, we propose a novel method named Rotary Position Embedding(RoPE) to effectively leverage the positional information. Specifically, the proposed RoPE encodes the absolute position with a rotation matrix and meanwhile incorporates the explicit relative position dependency in self-attention formulation. Notably, RoPE enables valuable properties, including the flexibility of sequence length, decaying inter-token dependency with increasing relative distances, and the capability of equipping the linear self-attention with relative position encoding. Finally, we evaluate the enhanced transformer with rotary position embedding, also called RoFormer, on various long text classification benchmark datasets. Our experiments show that it consistently overcomes its alternatives. Furthermore, we provide a theoretical analysis to explain some experimental results. RoFormer is already integrated into Huggingface: https://huggingface.co/docs/transformers/model_doc/roformer.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2104.09864")
get_code_for_paper("2104.09864")
have("2104.09864")
Connect an agent — have() is free.