We lifted 11 functions out of this paper's own repositories and ran 8 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| woosunghyeon/dropbp | canonical | 8 of 11 |
| Function | Status | Where it lives |
|---|---|---|
| build_tip_message | Ran | woosunghyeon/dropbp/huggingface/transformers_dropbp/utils/deprecate_models.py code served (permissive licence) · get_code("ab306165fa74656c") |
| get_config_class_from_processor_class | Ran | woosunghyeon/dropbp/huggingface/transformers_dropbp/utils/create_dummy_models.py code served (permissive licence) · get_code("3d999b95009c7537") |
| get_list_of_repo_model_paths | Ran | woosunghyeon/dropbp/huggingface/transformers_dropbp/utils/models_to_deprecate.py code served (permissive licence) · get_code("938fc7c2422ef91d") |
| get_model_doc_path | Ran | woosunghyeon/dropbp/huggingface/transformers_dropbp/utils/deprecate_models.py code served (permissive licence) · get_code("051d790d9b197d9d") |
| get_models | Ran | woosunghyeon/dropbp/huggingface/transformers_dropbp/utils/pr_slow_ci_models.py code served (permissive licence) · get_code("0dd0c5bdb6aa92c9") |
| get_new_python_files_between_commits | Ran | woosunghyeon/dropbp/huggingface/transformers_dropbp/utils/pr_slow_ci_models.py code served (permissive licence) · get_code("0f78c7037b35a137") |
| get_processor_types_from_config_class | Ran | woosunghyeon/dropbp/huggingface/transformers_dropbp/utils/create_dummy_models.py code served (permissive licence) · get_code("0c0b81b081919d8e") |
| parse_commit_message | Ran | woosunghyeon/dropbp/huggingface/transformers_dropbp/utils/pr_slow_ci_models.py code served (permissive licence) · get_code("17e5d8c36cc6bce8") |
| extract_model_info | Not yet run | woosunghyeon/dropbp/huggingface/transformers_dropbp/utils/deprecate_models.py code served (permissive licence) · get_code("54b7273383fcfc2d") |
| get_architectures_from_config_class | Not yet run | woosunghyeon/dropbp/huggingface/transformers_dropbp/utils/create_dummy_models.py code served (permissive licence) · get_code("b176efb26f72e507") |
| get_module_source_from_name | Not yet run | woosunghyeon/dropbp/huggingface/transformers_dropbp/utils/modular_model_converter.py code served (permissive licence) · get_code("93ccbb68485b6350") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Large language models (LLMs) have achieved significant success across various domains. However, training these LLMs typically involves substantial memory and computational costs during both forward and backward propagation. While parameter-efficient fine-tuning (PEFT) considerably reduces the training memory associated with parameters, it does not address the significant computational costs and activation memory. In this paper, we propose Dropping Backward Propagation (DropBP), a novel approach designed to reduce computational costs and activation memory while maintaining accuracy. DropBP randomly drops layers during backward propagation, which is essentially equivalent to training shallow submodules generated by undropped layers and residual connections. Additionally, DropBP calculates the sensitivity of each layer to assign an appropriate drop rate, thereby stabilizing the training process. DropBP is not only applicable to full fine-tuning but can also be orthogonally integrated with all types of PEFT by dropping layers during backward propagation. Specifically, DropBP can reduce training time by 44% with comparable accuracy to the baseline, accelerate convergence to the same perplexity by 1.5x, and enable training with a sequence length 6.2x larger on a single NVIDIA-A100 GPU. Furthermore, our DropBP enabled a throughput increase of 79% on a NVIDIA A100 GPU and 117% on an Intel Gaudi2 HPU. The code is available at https://github.com/WooSunghyeon/dropbp.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2402.17812")
get_code_for_paper("2402.17812")
have("2402.17812")
Connect an agent — have() is free.