Memory-Efficient LLM Training with Dynamic Sparsity: From Stability to Practical Scaling

Torsten Hoefler, Mykola Pechenizkiy, Decebal Mocanu, Elena Mocanu, Qiao Xiao, Boqian Wu, Maurice Van Keulen, Tomasz Sternal, Patrik Okanovic

arXiv · PDF · Open in the Atlas

Code that ran

We lifted 13 functions out of this paper's own repositories and ran 11 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.

Repositories linked to this paper

• QiaoXiao7282/SMET

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Dynamic Sparse Training (DST) offers a promising paradigm for improving the training and inference efficiency of deep neural networks; however, we find that in large language model training, DST can suffer from optimization instability, manifested as loss spikes after topology updates. In this work, we show that the naive use of standard Adam-based optimizers leads to a cold-start issue for newly regrown parameters, resulting in excessively large updates and disrupted training dynamics. To address this issue, we propose Sparse Memory-Efficient Training (SMET), which stabilizes DST with optimizer warm-up and improves training progress through density-aware learningrate scaling. SMET further reduces memory consumption by storing gradients and optimizer states only for active parameters. We provide a theoretical analysis of the update behaviors under SMET, showing improved optimization stability. Extensive experiments demonstrate that SMET enables stable, scalable, and memory-efficient sparse pretraining of LLMs, paving the way for sparse training as a practical alternative to dense training. Our code is publicly available at: https: //github.com/QiaoXiao7282/SMET.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2606.00888")
get_code_for_paper("2606.00888")
have("2606.00888")

Connect an agent — have() is free.