We lifted 15 functions out of this paper's own repositories and ran 13 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| chrisociepa/allamo | pwc_unofficial | 4 of 4 |
| lululuyi/longheads | extension | 3 of 4 |
| copy not recorded | — | 2 of 2 |
| yangjianxin1/longqlora | extension | 2 of 2 |
| hkunlp/chunkllama | extension | 1 of 2 |
| ymcui/chinese-llama-alpaca-2 | extension | 1 of 1 |
| Function | Status | Where it lives |
|---|---|---|
| apply_rotary_pos_emb | Ran | this paper's copy was not recorded; identical code first harvested from shaochenze/patchtrain pointer only · get_code("f725bc2d76076485") |
| apply_rotary_pos_emb_for_relative_keys | Ran | lululuyi/longheads/modeling_longheads/modeling_llama.py pointer only (licence: NONE) · get_code("adb0da12fb38486f") |
| calculate_md5 | Ran | chrisociepa/allamo/allamo/train_utils.py code served (permissive licence) · get_code("c168ab0101094c68") |
| estimate_mfu | Ran | chrisociepa/allamo/allamo/train_utils.py code served (permissive licence) · get_code("7cfe5dec1a04ab86") |
| evaluate | Ran | yangjianxin1/longqlora/script/evaluate/evaluate.py pointer only (licence: NONE) · get_code("24732fd7312ff4a8") |
| format_seconds_as_time | Ran | chrisociepa/allamo/allamo/train_utils.py code served (permissive licence) · get_code("8fa7d4f410dbb734") |
| generate_prompt | Ran | ymcui/chinese-llama-alpaca-2/scripts/inference/inference_hf.py code served (permissive licence) · get_code("b47f5a611055da10") |
| generate_prompt_landmark | Ran | lululuyi/longheads/passkey_retrieval/passkey_retrieval.py pointer only (licence: NONE) · get_code("2a07d4a8772a06b1") |
| get_fn_init_data | Ran | chrisociepa/allamo/allamo/model/lra.py code served (permissive licence) · get_code("94e2c02bdd3fbfa3") |
| get_mscale | Ran | hkunlp/chunkllama/chunkllama_attn_replace.py code served (permissive licence) · get_code("f530e017034693d2") |
| iceildiv | Ran | yangjianxin1/longqlora/script/evaluate/evaluate.py pointer only (licence: NONE) · get_code("51ee91de5e3fdcaa") |
| is_number_in_range | Ran | lululuyi/longheads/passkey_retrieval/passkey_retrieval.py pointer only (licence: NONE) · get_code("1a574afcd22201fe") |
| rotate_half | Ran | this paper's copy was not recorded; identical code first harvested from fe1ixxu/ALMA pointer only · get_code("b99eea6376d1e212") |
| merge_attn_outputs | Not yet run | hkunlp/chunkllama/chunkllama_attn_replace.py code served (permissive licence) · get_code("410511f95749dd69") |
| passkey_retrieval_test | Not yet run | lululuyi/longheads/passkey_retrieval/passkey_retrieval.py pointer only (licence: NONE) · get_code("41f1bbe0e5a8eaee") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
We present Position Interpolation (PI) that extends the context window sizes of RoPE-based pretrained LLMs such as LLaMA models to up to 32768 with minimal fine-tuning (within 1000 steps), while demonstrating strong empirical results on various tasks that require long context, including passkey retrieval, language modeling, and long document summarization from LLaMA 7B to 65B. Meanwhile, the extended model by Position Interpolation preserve quality relatively well on tasks within its original context window. To achieve this goal, Position Interpolation linearly down-scales the input position indices to match the original context window size, rather than extrapolating beyond the trained context length which may lead to catastrophically high attention scores that completely ruin the self-attention mechanism. Our theoretical study shows that the upper bound of interpolation is at least $\sim 600 \times$ smaller than that of extrapolation, further demonstrating its stability. Models extended via Position Interpolation retain its original architecture and can reuse most pre-existing optimization and infrastructure.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2306.15595")
get_code_for_paper("2306.15595")
have("2306.15595")
Connect an agent — have() is free.