Sung Hwang, Kaist, Korea, South Daejeon, Minki Kang, Moonsu Han
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
We propose a method to automatically generate a domain-and task-adaptive maskings of the given text for self-supervised pre-training, such that we can effectively adapt the language model to a particular target task (e.g. question answering). Specifically, we present a novel reinforcement learning-based framework which learns the masking policy, such that using the generated masks for further pre-training of the target language model helps improve task performance on unseen texts. We use off-policy actor-critic with entropy regularization and experience replay for reinforcement learning, and propose a Transformer-based policy network that can consider the relative importance of words in a given text. We validate our Neural Mask Generator (NMG) on several question answering and text classification datasets using BERT and DistilBERT as the language models, on which it outperforms rule-based masking strategies, by automatically learning optimal adaptive maskings. 1
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2010.02705")
get_code_for_paper("2010.02705")
have("2010.02705")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2010.02705.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2010.02705)
Connect an agent — have() is free.