SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2407.02486 · NAACL · 2024

Neurocache: Efficient Vector Retrieval for Long-range Language Modeling

Deniz Yuret, Ali Safaya

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

This paper introduces Neurocache, an approach to extend the effective context size of large language models (LLMs) using an external vector cache to store its past states. Like recent vector retrieval approaches, Neurocache uses an efficient k-nearest-neighbor (kNN) algorithm to retrieve relevant past states and incorporate them into the attention process. Neurocache improves upon previous methods by (1) storing compressed states, which reduces cache size; (2) performing a single retrieval operation per token which increases inference speed; and (3) extending the retrieval window to neighboring states, which improves both language modeling and downstream task accuracy. Our experiments show the effectiveness of Neurocache both for models trained from scratch and for pre-trained models such as Llama2-7B and Mistral-7B when enhanced with the cache mechanism. We also compare Neurocache with text retrieval methods and show improvements in single-document questionanswering and few-shot learning tasks. We made the source code available under:

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2407.02486")
get_code_for_paper("2407.02486")
have("2407.02486")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2407.02486.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2407.02486.svg)](https://syntology.ai/paper/2407.02486)

Connect an agent — have() is free.