Junho Kim, Kang-Min Kim, Jun-Hyung Park, Sangkeun Lee, Mingyu Lee
We lifted 1 functions out of this paper's own repositories and ran 1 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| koreamglee/concept-based-curriculum-masking | — | 1 of 1 |
| Function | Status | Where it lives |
|---|---|---|
| construct_next_stage | Ran | koreamglee/concept-based-curriculum-masking/script/curriculum_construction.py pointer only (licence: NONE) · get_code("ee7739070278126e") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Masked language modeling (MLM) has been widely used for pre-training effective bidirectional representations, but incurs substantial training costs. In this paper, we propose a novel concept-based curriculum masking (CCM) method to efficiently pre-train a language model. CCM has two key differences from existing curriculum learning approaches to effectively reflect the nature of MLM. First, we introduce a carefully-designed linguistic difficulty criterion that evaluates the MLM difficulty of each token. Second, we construct a curriculum that gradually masks words related to the previously masked words by retrieving a knowledge graph. Experimental results show that CCM significantly improves pre-training efficiency. Specifically, the model trained with CCM shows comparative performance with the original BERT on the General Language Understanding Evaluation benchmark at half of the training cost. Code is available at https://github.com/KoreaMGLEE/Conceptbased-curriculum-masking.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2212.07617")
get_code_for_paper("2212.07617")
have("2212.07617")
Connect an agent — have() is free.