Bo Zheng, Tong Xiao, Jingbo Zhu, Wenbo Su, Haibin Chen, Jiwei Tang, Weidong Zhang, Shilei Liu, Runsong Zhao, Yujin Yuan, Langming Liu
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
The quadratic complexity and indefinitely growing key-value (KV) cache of standard Transformers pose a major barrier to longcontext processing. To overcome this, we introduce the Collaborative Memory Transformer (CoMeT), a novel architecture that enables LLMs to handle arbitrarily long sequences with constant memory usage and linear time complexity. Designed as an efficient, plug-in module, CoMeT can be integrated into pre-trained models with only minimal fine-tuning. It operates on sequential data chunks, using a dualmemory system to manage context: a temporary memory on a FIFO queue for recent events, and a global memory with a gated update rule for long-range dependencies. These memories then act as a dynamic soft prompt for the next chunk. To enable efficient finetuning on extremely long contexts, we introduce a novel layer-level pipeline parallelism strategy. The effectiveness of our approach is remarkable: a model equipped with CoMeT and fine-tuned on 32k contexts can accurately retrieve a passkey from any position within a 1M token sequence. On the SCROLLS benchmark, CoMeT surpasses other efficient methods and achieves performance comparable to a fullattention baseline on summarization tasks. Its practical effectiveness is further validated on real-world agent and user behavior QA tasks.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2602.01766")
get_code_for_paper("2602.01766")
have("2602.01766")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2602.01766.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2602.01766)
Connect an agent — have() is free.