Wotao Yin, Xinmeng Huang, Kun Yuan, Yiming Chen, Yingya Zhang, Pan Pan, Yinghui Xu
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
The scale of deep learning nowadays calls for efficient distributed training algorithms. Decentralized momentum SGD (DmSGD), in which each node averages only with its neighbors, is more communication efficient than vanilla Parallel momentum SGD that incurs global average across all computing nodes. On the other hand, the large-batch training has been demonstrated critical to achieve runtime speedup. This motivates us to investigate how DmSGD performs in the large-batch scenario. In this work, we find the momentum term can amplify the inconsistency bias in DmSGD. Such bias becomes more evident as batch-size grows large and hence results in severe performance degradation. We next propose DecentLaM, a novel decentralized large-batch momentum SGD to remove the momentum-incurred bias. The convergence rate for both strongly convex and non-convex scenarios is established. Our theoretical results justify the superiority of DecentLaM to DmSGD especially in the large-batch scenario. Experimental results on a a variety of computer vision tasks and models show that DecentLaM promises both efficient and high-quality training.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2104.11981")
get_code_for_paper("2104.11981")
have("2104.11981")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2104.11981.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2104.11981)
Connect an agent — have() is free.