MINTEVAL: Evaluating Memory under Multi-Target Interference in Long-Horizon Agent Systems

Mohit Bansal, Zaid Khan, Elias Stengel-Eskin, Justin Chih-Yao Chen, Hyunji Lee, Joykirat Singh

arXiv · PDF · Open in the Atlas

Code that ran

We lifted 6 functions out of this paper's own repositories and ran 6 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.

Repositories linked to this paper

• amy-hyunji/MINTEval

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Agents in real-world settings operate over long and evolving horizons, where information is repeatedly updated and may interfere across memories, requiring accurate recall and aggregated reasoning over multiple pieces of information. However, existing benchmarks focus on static, independent recall and fail to capture these dynamic interactions between evolving memories. In this paper, we study how current memory-augmented agents perform in realistic, interference-heavy, long-horizon settings across diverse domains and question types. To this end, we introduce MINTEVAL (Long-Horizon Memory under INTerference Evaluation), an analytical benchmark which features (1) long, highly interconnected contexts with frequently updated information that induces substantial interference, (2) diverse domains (state tracking, multi-turn dialogue, Wikipedia revisions, and GitHub commits), enabling evaluation of domain generalization, and (3) diverse question types that assess robustness to interference, including (i) single-target recall tasks requiring retrieval of a specific target from long contexts, and (ii) multi-target aggregation tasks requiring reasoning over multiple relevant pieces of information. Overall, MINTEVAL contains 15.6k question-answering pairs over long-horizon contexts averaging 138.8k tokens and extending up to 1.8M tokens per instance. We evaluate seven representative systems, including vanilla long-context LLMs, retrievalaugmented generation methods, and memory-augmented agent frameworks. Across all systems, we observe consistently low performance (avg. 27.9% accuracy), especially on questions requiring aggregated reasoning over multiple pieces of evidence. Fine-grained analysis shows that performance is primarily limited by retrieval and memory construction capabilities. Furthermore, current memory systems struggle to recall and reason over earlier facts that are later revised or interfered with by subsequent context, with performance degrading as the number of intervening updates increases. These findings highlight the need for more robust memory management systems for dynamic, long-horizon environments across varying domains. Code and data are available at https://github.com/amy-hyunji/MINTEval.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2605.18565")
get_code_for_paper("2605.18565")
have("2605.18565")

Connect an agent — have() is free.