SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2602.01766 · 2026

CoMeT: Collaborative Memory Transformer for Efficient Long Context Modeling

Bo Zheng, Tong Xiao, Jingbo Zhu, Wenbo Su, Haibin Chen, Jiwei Tang, Weidong Zhang, Shilei Liu, Runsong Zhao, Yujin Yuan, Langming Liu

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

The quadratic complexity and indefinitely growing key-value (KV) cache of standard Transformers pose a major barrier to longcontext processing. To overcome this, we introduce the Collaborative Memory Transformer (CoMeT), a novel architecture that enables LLMs to handle arbitrarily long sequences with constant memory usage and linear time complexity. Designed as an efficient, plug-in module, CoMeT can be integrated into pre-trained models with only minimal fine-tuning. It operates on sequential data chunks, using a dualmemory system to manage context: a temporary memory on a FIFO queue for recent events, and a global memory with a gated update rule for long-range dependencies. These memories then act as a dynamic soft prompt for the next chunk. To enable efficient finetuning on extremely long contexts, we introduce a novel layer-level pipeline parallelism strategy. The effectiveness of our approach is remarkable: a model equipped with CoMeT and fine-tuned on 32k contexts can accurately retrieve a passkey from any position within a 1M token sequence. On the SCROLLS benchmark, CoMeT surpasses other efficient methods and achieves performance comparable to a fullattention baseline on summarization tasks. Its practical effectiveness is further validated on real-world agent and user behavior QA tasks.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2602.01766")
get_code_for_paper("2602.01766")
have("2602.01766")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2602.01766.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2602.01766.svg)](https://syntology.ai/paper/2602.01766)

Connect an agent — have() is free.