We lifted 2 functions out of this paper's own repositories and ran 2 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| sail-sg/lm-random-memory-access | canonical | 2 of 2 |
| Function | Status | Where it lives |
|---|---|---|
| prefix_match_em_score | Ran | sail-sg/lm-random-memory-access/trainer_gpt_qa.py pointer only (licence: NONE) · get_code("feb77809d6acd3aa") |
| remove_prefix | Ran | sail-sg/lm-random-memory-access/trainer_gpt_qa.py pointer only (licence: NONE) · get_code("667158d196f22872") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Recent developments in Language Models (LMs) have shown their effectiveness in NLP tasks, particularly in knowledge-intensive tasks. However, the mechanisms underlying knowledge storage and memory access within their parameters remain elusive. In this paper, we investigate whether a generative LM (e.g., GPT-2) is able to access its memory sequentially or randomly. Through carefully-designed synthetic tasks, covering the scenarios of full recitation, selective recitation and grounded question answering, we reveal that LMs manage to sequentially access their memory while encountering challenges in randomly accessing memorized content. We find that techniques including recitation and permutation improve the random memory access capability of LMs. Furthermore, by applying this intervention to realistic scenarios of open-domain question answering, we validate that enhancing random access by recitation leads to notable improvements in question answering. The code to reproduce our experiments can be found at https://github.com/sail-sg/lm-random-memory-access.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2403.07805")
get_code_for_paper("2403.07805")
have("2403.07805")
Connect an agent — have() is free.