Bryan Catanzaro, William Hamilton, Wei Ping, Devendra Singh Sachan, Mohammad Shoeybi, Mostofa Patwary, Devendra Sachan, Neel Kant
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. If it links a repository, it is listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
open-domain question answering (OpenQA) has employed both supervised and unsupervised approaches. However, it remains unclear how unsupervised and supervised methods can be used most effectively for neural retrievers. In this work, we systematically study retriever pre-training. We first propose an approach of unsupervised pre-training with the Inverse Cloze Task and masked salient spans, followed by supervised finetuning using question-context pairs. This approach leads to absolute gains of 2+ points over the previous best result in the top-20 retrieval accuracy on Natural Questions and TriviaQA datasets. We next explore two approaches for end-toend training of the reader and retriever components in OpenQA models, which differ in the manner the reader ingests the retrieved documents. Our experiments demonstrate the effectiveness of these approaches as we obtain state-of-the-art results. On the Natural Questions dataset, we obtain a top-20 retrieval accuracy of 84%, an improvement of 5 points over the recent DPR model. We also showcase good results on answer extraction, outperforming recent models such as REALM and RAG by 3+ points. Our code is available at: https: //github.com/NVIDIA/Megatron-LM.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2101.00408")
get_code_for_paper("2101.00408")
have("2101.00408")
Connect an agent — have() is free.