We lifted 7 functions out of this paper's own repositories and ran 5 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| bui1dmysea/memlong | canonical | 5 of 7 |
| Function | Status | Where it lives |
|---|---|---|
| repeat_kv | Ran | bui1dmysea/memlong/eval/icl/MemLong/modeling_llama.py pointer only (licence: NONE) · get_code("30d7eec482ebf6b1") |
| apply_rotary_pos_emb | Ran | bui1dmysea/memlong/eval/icl/MemLong/modeling_llama.py pointer only (licence: NONE) · get_code("bac65c3dafaec040") |
| group_texts | Ran | bui1dmysea/memlong/eval/language_modeling/MemLong/utils.py pointer only (licence: NONE) · get_code("65bc466999b923d6") |
| rotate_half | Ran | bui1dmysea/memlong/eval/icl/MemLong/modeling_llama.py pointer only (licence: NONE) · get_code("b99eea6376d1e212") |
| tokenize_fn | Ran | bui1dmysea/memlong/eval/language_modeling/MemLong/utils.py pointer only (licence: NONE) · get_code("c78930a2d9e9be54") |
| apply_rotary_pos_emb_for_relative_query | Not yet run | bui1dmysea/memlong/eval/icl/MemLong/modeling_llama_position.py pointer only (licence: NONE) · get_code("7ce77847b8f6ae5c") |
| convert_to_lora | Not yet run | bui1dmysea/memlong/eval/language_modeling/MemLong/utils.py pointer only (licence: NONE) · get_code("187d8d92b86625bd") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Recent advancements in Large Language Models (LLMs) have yielded remarkable success across diverse fields. However, handling long contexts remains a significant challenge for LLMs due to the quadratic time and space complexity of attention mechanisms and the growing memory consumption of the key-value cache during generation. This work introduces MemLong: Memory-Augmented Retrieval for Long Text Generation, a method designed to enhance the capabilities of long-context language modeling by utilizing an external retriever for historical information retrieval. MemLong combines a non-differentiable ``ret-mem'' module with a partially trainable decoder-only language model and introduces a fine-grained, controllable retrieval attention mechanism that leverages semantic-level relevant chunks. Comprehensive evaluations on multiple long-context language modeling benchmarks demonstrate that MemLong consistently outperforms other state-of-the-art LLMs. More importantly, MemLong can extend the context length on a single 3090 GPU from 4k up to 80k. Our code is available at https://github.com/Bui1dMySea/MemLong
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2408.16967")
get_code_for_paper("2408.16967")
have("2408.16967")
Connect an agent — have() is free.