We lifted 1 functions out of this paper's own repositories and ran 0 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| apoorvkh/academic-pretraining | canonical | 0 of 1 |
| Function | Status | Where it lives |
|---|---|---|
| ampere_or_newer_gpu | Not yet run | apoorvkh/academic-pretraining/src/gpus.py code served (permissive licence) · get_code("028e2556b15a8837") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Pre-training is notoriously compute-intensive and academic researchers are notoriously under-resourced. It is, therefore, commonly assumed that academics can't pre-train models. In this paper, we seek to clarify this assumption. We first survey academic researchers to learn about their available compute and then empirically measure the time to replicate models on such resources. We introduce a benchmark to measure the time to pre-train models on given GPUs and also identify ideal settings for maximizing training speed. We run our benchmark on a range of models and academic GPUs, spending 2,000 GPU-hours on our experiments. Our results reveal a brighter picture for academic pre-training: for example, although Pythia-1B was originally trained on 64 GPUs for 3 days, we find it is also possible to replicate this model (with the same hyper-parameters) in 3x fewer GPU-days: i.e. on 4 GPUs in 18 days. We conclude with a cost-benefit analysis to help clarify the trade-offs between price and pre-training time. We believe our benchmark will help academic researchers conduct experiments that require training larger models on more data. We fully release our codebase at: https://github.com/apoorvkh/academic-pretraining.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2410.23261")
get_code_for_paper("2410.23261")
have("2410.23261")
Connect an agent — have() is free.