Dinh Phung, Gholamreza Haffari, Thuy-Trang Vu
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Recent work has shown the importance of adaptation of broad-coverage contextualised embedding models on the domain of the target task of interest. Current self-supervised adaptation methods are simplistic, as the training signal comes from a small percentage of randomly masked-out tokens. In this paper, we show that careful masking strategies can bridge the knowledge gap of masked language models (MLMs) about the domains more effectively by allocating self-supervision where it is needed. Furthermore, we propose an effective training strategy by adversarially masking out those tokens which are harder to reconstruct by the underlying MLM. The adversarial objective leads to a challenging combinatorial optimisation problem over subsets of tokens, which we tackle efficiently through relaxation to a variational lower-bound and dynamic programming. On six unsupervised domain adaptation tasks involving named entity recognition, our method strongly outperforms the random masking strategy and achieves up to +1.64 F1 score improvements.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2010.01739")
get_code_for_paper("2010.01739")
have("2010.01739")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2010.01739.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2010.01739)
Connect an agent — have() is free.