Bo Li, Dahua Lin, Pengfei Liu, Ruihao Gong, Xin Jin, Yazhe Niu, Yongqiang Yao, Jingru Tan, Feizhao Zhang, Jiahao Hu, Ningyi Xu
We lifted 5 functions out of this paper's own repositories and ran 3 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| modeltc/omnibal | — | 3 of 5 |
| Function | Status | Where it lives |
|---|---|---|
| get_sp_groups | Ran | modeltc/omnibal/dataset.py code served (permissive licence) · get_code("bca40cc122c9e2c2") |
| get_token_sum | Ran | modeltc/omnibal/dataset.py code served (permissive licence) · get_code("247c4331ff8ffd71") |
| get_vit_num | Ran | modeltc/omnibal/dataset.py code served (permissive licence) · get_code("4ec67a8cb2818342") |
| BalancedDataset | Not yet run | modeltc/omnibal/dataset.py code served (permissive licence) · get_code("b95b1c76bb746221") |
| get_sp_dist_pad_ratio | Not yet run | modeltc/omnibal/dataset.py code served (permissive licence) · get_code("55ca6282c82aa63a") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Vision-language instruction-tuning models have recently achieved significant performance improvements. In this work, we discover that largescale 3D parallel training on those models leads to an imbalanced computation load across different devices. The vision and language parts are inherently heterogeneous: their data distribution and model architecture differ significantly, which affects distributed training efficiency. To address this issue, we rebalance the computational load from data, model, and memory perspectives, achieving more balanced computation across devices. Specifically, for the data, instances are grouped into new balanced mini-batches within and across devices. A search-based method is employed for the model to achieve a more balanced partitioning. For memory optimization, we adaptively adjust the re-computation strategy for each partition to utilize the available memory fully. These three perspectives are not independent but are closely connected, forming an omniverse balanced training framework. Extensive experiments are conducted to validate the effectiveness of our method. Compared with the open-source training code of InternVL-Chat, training time is reduced greatly, achieving about 1.8× speed-up. Our method's efficacy and generalizability are further validated across various models and datasets. Codes will be released at https://github.com/ModelTC/OmniBal.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2407.20761")
get_code_for_paper("2407.20761")
have("2407.20761")
Connect an agent — have() is free.