Junxian He, Graham Neubig, Uri Alon, Frank Xu, Dan Roth, Sudipta Sengupta
We lifted 4 functions out of this paper's own repositories and ran 1 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| neulab/retomaton | canonical | 0 of 1 |
| copy not recorded | — | 1 of 1 |
| neulab/knn-transformers | — | 0 of 2 |
| Function | Status | Where it lives |
|---|---|---|
| get_members_path | Ran | this paper's copy was not recorded; identical code first harvested from neulab/knn-transformers pointer only · get_code("5b87ede004fa49b0") |
| KNNWrapper | Not yet run | neulab/knn-transformers/retomaton.py code served (permissive licence) · get_code("ce70a7f3d1a52dba") |
| KNN_Dstore | Not yet run | neulab/retomaton/fairseq/knnlm.py code served (permissive licence) · get_code("0ebdfa0124e8ed1a") |
| RetomatonWrapper | Not yet run | neulab/knn-transformers/retomaton.py code served (permissive licence) · get_code("616b7b6be663b3a6") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Retrieval-based language models (R-LM) model the probability of natural language text by combining a standard language model (LM) with examples retrieved from an external datastore at test time. While effective, a major bottleneck of using these models in practice is the computationally costly datastore search, which can be performed as frequently as every time step. In this paper, we present RETOMATON -retrieval automaton -which approximates the datastore search, based on (1) saving pointers between consecutive datastore entries, and (2) clustering of entries into "states". This effectively results in a weighted finite automaton built on top of the datastore, instead of representing the datastore as a flat list. The creation of the automaton is unsupervised, and a RETOMATON can be constructed from any text collection: either the original training corpus or from another domain. Traversing this automaton at inference time, in parallel to the LM inference, reduces its perplexity by up to 1.85, or alternatively saves up to 83% of the nearest neighbor searches over kNN-LM (Khandelwal et al., 2020) without hurting perplexity. Our code and trained models are available at https: //github.com/neulab/retomaton .
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2201.12431")
get_code_for_paper("2201.12431")
have("2201.12431")
Connect an agent — have() is free.