We lifted 5 functions out of this paper's own repositories and ran 2 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| OswaldHe/HMT-pytorch | canonical | 2 of 5 |
| Function | Status | Where it lives |
|---|---|---|
| PubMedQA | Ran | OswaldHe/HMT-pytorch/hmt_src/pubmedqa_ds_preprocess.py code served (permissive licence) · get_code("73a56369878e9809") |
| inject_eae | Ran | OswaldHe/HMT-pytorch/modeling_rmt/compression.py code served (permissive licence) · get_code("acb3090c9c09b349") |
| LongSFT | Not yet run | OswaldHe/HMT-pytorch/hmt_src/long_sft_ds_preprocess.py code served (permissive licence) · get_code("ea22b08819eb2608") |
| OpenROAD | Not yet run | OswaldHe/HMT-pytorch/hmt_src/openroad_qa_preprocess.py code served (permissive licence) · get_code("386041385855d2e1") |
| OpenROAD_test | Not yet run | OswaldHe/HMT-pytorch/hmt_src/openroad_qa_preprocess.py code served (permissive licence) · get_code("5ae82ad3d97c3278") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Transformer-based large language models (LLM) have been widely used in language processing applications. However, due to the memory constraints of the devices, most of them restrict the context window. Even though recurrent models in previous works can memorize past tokens to enable unlimited context and maintain effectiveness, they have ``flat'' memory architectures. Such architectures have limitations in selecting and filtering information. Since humans are good at learning and self-adjustment, we believe that imitating brain memory hierarchy is beneficial for model memorization. Thus, we propose the Hierarchical Memory Transformer (HMT), a novel framework that facilitates a model's long-context processing ability by imitating human memorization behavior. Leveraging memory-augmented segment-level recurrence, we organize the memory hierarchy by preserving tokens from early input segments, passing memory embeddings along the sequence, and recalling relevant information from history. Evaluating general language modeling, question-answering tasks, and the summarization task, we show that HMT consistently improves the long-context processing ability of existing models. Furthermore, HMT achieves a comparable or superior generation quality to long-context LLMs with $2 \sim 57\times$ fewer parameters and $2.5 \sim 116\times$ less inference memory, significantly outperforming previous memory-augmented models. Code on Github: https://github.com/OswaldHe/HMT-pytorch.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2405.06067")
get_code_for_paper("2405.06067")
have("2405.06067")
Connect an agent — have() is free.