We lifted 11 functions out of this paper's own repositories and ran 9 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| qhjqhj00/memorag | canonical | 9 of 11 |
| Function | Status | Where it lives |
|---|---|---|
| cat_tensor | Ran | qhjqhj00/memorag/train/src/modeling_beacon.py code served (permissive licence) · get_code("019064fb2dc127e6") |
| except_retry_dec | Ran | qhjqhj00/memorag/memorag/agent.py code served (permissive licence) · get_code("8f2480ff6950770e") |
| get_shifted_labels | Ran | qhjqhj00/memorag/train/src/modeling_utils.py code served (permissive licence) · get_code("5e960b1ca7bef007") |
| mask_nested_lists | Ran | qhjqhj00/memorag/train/src/chat.py code served (permissive licence) · get_code("141f4c8b387bca52") |
| merge_inputs | Ran | qhjqhj00/memorag/memorag/memorag.py code served (permissive licence) · get_code("f29d09a2b245b362") |
| move_to_device | Ran | qhjqhj00/memorag/train/src/modeling_utils.py code served (permissive licence) · get_code("b8a81a4b39c05cff") |
| optional_grad_ctx | Ran | qhjqhj00/memorag/train/src/modeling_utils.py code served (permissive licence) · get_code("7598dbd722b803a3") |
| slice_activations | Ran | qhjqhj00/memorag/train/src/modeling_beacon.py code served (permissive licence) · get_code("defaf52f8cef4a26") |
| slice_tensor | Ran | qhjqhj00/memorag/train/src/modeling_beacon.py code served (permissive licence) · get_code("6b7f09c9bef8e516") |
| apply_chat_template | Not yet run | qhjqhj00/memorag/train/src/chat.py code served (permissive licence) · get_code("561fab9ba99c10a9") |
| get_conv_template | Not yet run | qhjqhj00/memorag/train/src/chat.py code served (permissive licence) · get_code("63d36f1757a0760e") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Processing long contexts presents a significant challenge for large language models (LLMs). While recent advancements allow LLMs to handle much longer contexts than before (e.g., 32K or 128K tokens), it is computationally expensive and can still be insufficient for many applications. Retrieval-Augmented Generation (RAG) is considered a promising strategy to address this problem. However, conventional RAG methods face inherent limitations because of two underlying requirements: 1) explicitly stated queries, and 2) well-structured knowledge. These conditions, however, do not hold in general long-context processing tasks. In this work, we propose MemoRAG, a novel RAG framework empowered by global memory-augmented retrieval. MemoRAG features a dual-system architecture. First, it employs a light but long-range system to create a global memory of the long context. Once a task is presented, it generates draft answers, providing useful clues for the retrieval tools to locate relevant information within the long context. Second, it leverages an expensive but expressive system, which generates the final answer based on the retrieved information. Building upon this fundamental framework, we realize the memory module in the form of KV compression, and reinforce its memorization and cluing capacity from the Generation quality's Feedback (a.k.a. RLGF). In our experiments, MemoRAG achieves superior performances across a variety of long-context evaluation tasks, not only complex scenarios where traditional RAG methods struggle, but also simpler ones where RAG is typically applied.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2409.05591")
get_code_for_paper("2409.05591")
have("2409.05591")
Connect an agent — have() is free.