Cho-Jui Hsieh, U Berkeley, Sanjiv Kumar, Kurt Keutzer, Sashank Reddi, Xiaodan Song, Srinadh Bhojanapalli, Yang You, Jing Li, James Demmel, Google Google, Jonathan Hseu
We lifted 11 functions out of this paper's own repositories and ran 6 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| meltnur/speed | pwc_unofficial | 4 of 9 |
| bojone/tiger | pwc_unofficial | 2 of 2 |
| Function | Status | Where it lives |
|---|---|---|
| get_lr_from_config | Ran | meltnur/speed/deepspeed/pt/deepspeed_lr_schedules.py pointer only (licence: MIT) · get_code("97777449b62c0a36") |
| get_scalar_param | Ran | meltnur/speed/deepspeed/pt/deepspeed_config.py pointer only (licence: MIT) · get_code("8ecbd0d0bd8ec80f") |
| parse_resource_filter | Ran | meltnur/speed/deepspeed/pt/deepspeed_run.py pointer only (licence: MIT) · get_code("77529b2a36ec2e49") |
| piecewise_linear | Ran | bojone/tiger/tiger.py code served (permissive licence) · get_code("6a8666be8c9e0af6") |
| root_mean_square | Ran | bojone/tiger/tiger.py code served (permissive licence) · get_code("7ced0d41eac533bc") |
| to_python_float | Ran | meltnur/speed/deepspeed/pt/loss_scaler.py pointer only (licence: MIT) · get_code("6b5d363ab4dbf012") |
| add_tuning_arguments | Not yet run | meltnur/speed/deepspeed/pt/deepspeed_lr_schedules.py pointer only (licence: MIT) · get_code("a9c0a909a533ca74") |
| fetch_hostfile | Not yet run | meltnur/speed/deepspeed/pt/deepspeed_run.py pointer only (licence: MIT) · get_code("532b9a790bd9609b") |
| get_config_from_args | Not yet run | meltnur/speed/deepspeed/pt/deepspeed_lr_schedules.py pointer only (licence: MIT) · get_code("6e89a38d49e658cc") |
| get_fp16_enabled | Not yet run | meltnur/speed/deepspeed/pt/deepspeed_config.py pointer only (licence: MIT) · get_code("93f7995d74994b25") |
| get_loss_scale | Not yet run | meltnur/speed/deepspeed/pt/deepspeed_config.py pointer only (licence: MIT) · get_code("3c0009bff557332d") |
More are linked than shown here.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Training large deep neural networks on massive datasets is computationally very challenging. There has been recent surge in interest in using large batch stochastic optimization methods to tackle this issue. The most prominent algorithm in this line of research is LARS, which by employing layerwise adaptive learning rates trains RESNET on ImageNet in a few minutes. However, LARS performs poorly for attention models like BERT, indicating that its performance gains are not consistent across tasks. In this paper, we first study a principled layerwise adaptation strategy to accelerate training of deep neural networks using large mini-batches. Using this strategy, we develop a new layerwise adaptive large batch optimization technique called LAMB; we then provide convergence analysis of LAMB as well as LARS, showing convergence to a stationary point in general nonconvex settings. Our empirical results demonstrate the superior performance of LAMB across various tasks such as BERT and RESNET-50 training with very little hyperparameter tuning. In particular, for BERT training, our optimizer enables use of very large batch sizes of 32868 without any degradation of performance. By increasing the batch size to the memory limit of a TPUv3 Pod, BERT training time can be reduced from 3 days to just 76 minutes (Table 1). The LAMB implementation is available online 1 .
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("1904.00962")
get_code_for_paper("1904.00962")
have("1904.00962")
Connect an agent — have() is free.