Maosong Sun, Zhiyuan Liu, Jie Zhou, Xu Han, Yankai Lin, Zhiyuan Zeng, Ruobing Xie, Zhengyan Zhang, Chaojun Xiao, Qiujieli Qin
We lifted 4 functions out of this paper's own repositories and ran 4 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| thunlp/moefication | canonical | 4 of 4 |
| Function | Status | Where it lives |
|---|---|---|
| cal_intertia_ | Ran | thunlp/moefication/faster_moefication/moefication.py pointer only (licence: NONE) · get_code("703e3ecf17cfa606") |
| get_layer_num | Ran | thunlp/moefication/moefication/utils.py pointer only (licence: NONE) · get_code("0a1340701fa70104") |
| load_ffn_weight | Ran | thunlp/moefication/moefication/utils.py pointer only (licence: NONE) · get_code("158405f0f4beeff8") |
| load_hidden_states | Ran | thunlp/moefication/moefication/utils.py pointer only (licence: NONE) · get_code("ceb1bef670471d4e") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Pre-trained Transformers inherently possess the characteristic of sparse activation, where only a small fraction of the neurons are activated for each token. While sparse activation has been explored through post-training methods, its potential in pre-training remains untapped. In this work, we first study how activation properties change during pre-training. Our examination reveals that Transformers exhibit sparse activation throughout the majority of the pretraining process while the activation correlation keeps evolving as training progresses. Leveraging this observation, we propose Switchable Sparse-Dense Learning (SSD). SSD adaptively switches between the Mixtures-of-Experts (MoE) based sparse training and the conventional dense training during the pre-training process, leveraging the efficiency of sparse training and avoiding the static activation correlation of sparse training. Compared to dense training, SSD achieves comparable performance with identical model size and reduces pre-training costs. Moreover, the models trained with SSD can be directly used as MoE models for sparse inference and achieve the same performance as dense models with up to 2× faster inference speed. Codes are available at https: //github.com/thunlp/moefication.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2410.03440")
get_code_for_paper("2410.03440")
have("2410.03440")
Connect an agent — have() is free.