Wulong Liu, Jianye Hao, Chen Chen, Bin Li, Jie Wang, Lei Chen, Mingxuan Yuan, Xing Li, Xianzhi Yu, Qingyue Yang, Zhihai Wang
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
With the development of large language models (LLMs), efficient inference through Key-Value (KV) cache compression has attracted considerable attention, especially for long-context generation. To compress the KV cache, recent methods identify critical KV tokens through static modeling of attention scores. However, these methods often struggle to accurately determine critical tokens as they neglect the temporal patterns in attention scores, resulting in a noticeable degradation in LLM performance. To address this challenge, we propose AttentionPredictor, which is the first learning-based method to directly predict attention patterns for KV cache compression and critical token identification. Specifically, Attention-Predictor learns a lightweight, unified convolution model to dynamically capture spatiotemporal patterns and predict the next-token attention scores. An appealing feature of AttentionPredictor is that it accurately predicts the attention score and shares the unified prediction model, which consumes negligible memory, among all transformer layers. Moreover, we propose a cross-token critical cache prefetching framework that hides the token estimation time overhead to accelerate the decoding stage. By retaining most of the attention information, AttentionPredictor achieves 13× KV cache compression and 5.6× speedup in a cache offloading scenario with comparable LLM performance, significantly outperforming the stateof-the-arts. The code is available at https://github.com/MIRALab-USTC/LLM-AttentionPredictor.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2502.04077")
get_code_for_paper("2502.04077")
have("2502.04077")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2502.04077.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2502.04077)
Connect an agent — have() is free.