We lifted 5 functions out of this paper's own repositories and ran 5 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| trustedllm/regavae | canonical | 5 of 5 |
| Function | Status | Where it lives |
|---|---|---|
| calc_au | Ran | trustedllm/regavae/Step1/train_utils.py pointer only (licence: NONE) · get_code("2887245599da5a9d") |
| calc_iwnll | Ran | trustedllm/regavae/Step1/train_utils.py pointer only (licence: NONE) · get_code("046dedc52d7678ab") |
| calc_iwnll | Ran | trustedllm/regavae/Step2/train_utils.py pointer only (licence: NONE) · get_code("da09f461070b46d3") |
| log_sum_exp | Ran | trustedllm/regavae/Step1/train_utils.py pointer only (licence: NONE) · get_code("5efe6eb2209f6591") |
| soft_clamp5 | Ran | trustedllm/regavae/Step1/dist.py pointer only (licence: NONE) · get_code("cd334034d398bfaa") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Retrieval-augmented language models show promise in addressing issues like outdated information and hallucinations in language models (LMs). However, current research faces two main problems: 1) determining what information to retrieve, and 2) effectively combining retrieved information during generation. We argue that valuable retrieved information should not only be related to the current source text but also consider the future target text, given the nature of LMs that model future tokens. Moreover, we propose that aggregation using latent variables derived from a compact latent space is more efficient than utilizing explicit raw text, which is limited by context length and susceptible to noise. Therefore, we introduce RegaVAE, a retrieval-augmented language model built upon the variational auto-encoder (VAE). It encodes the text corpus into a latent space, capturing current and future information from both source and target text. Additionally, we leverage the VAE to initialize the latent space and adopt the probabilistic form of the retrieval generation paradigm by expanding the Gaussian prior distribution into a Gaussian mixture distribution. Theoretical analysis provides an optimizable upper bound for RegaVAE. Experimental results on various datasets demonstrate significant improvements in text generation quality and hallucination removal.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2310.10567")
get_code_for_paper("2310.10567")
have("2310.10567")
Connect an agent — have() is free.