Ping Luo, Hong Kong, Qian Liu, Min Lin, Chaofan Tao, Zhongwei Wan, Niklas Muennighoff, Ngai Wong, Longxu Dou, Contextual Ai, Sea Lab
We lifted 6 functions out of this paper's own repositories and ran 5 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| sail-sg/scaling-with-vocab | canonical | 5 of 6 |
| Function | Status | Where it lives |
|---|---|---|
| LSE | Ran | sail-sg/scaling-with-vocab/approach1_isoflops.py pointer only (licence: NONE) · get_code("c8ff0a0bc420dacd") |
| LSE | Ran | sail-sg/scaling-with-vocab/approach2_derivative.py pointer only (licence: NONE) · get_code("fe98406b87b34d45") |
| LSE_Nnv_H | Ran | sail-sg/scaling-with-vocab/approach1_isoflops.py pointer only (licence: NONE) · get_code("8031b85f41bc755f") |
| dF_dV | Ran | sail-sg/scaling-with-vocab/approach2_derivative.py pointer only (licence: NONE) · get_code("efa2370bd36a4ee6") |
| objective_function_Nnv | Ran | sail-sg/scaling-with-vocab/approach2_derivative.py pointer only (licence: NONE) · get_code("bccf520fe6c78f34") |
| dl_dv | Not yet run | sail-sg/scaling-with-vocab/approach3_isoloss.py pointer only (licence: NONE) · get_code("fd48483acd7fe2c1") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Research on scaling large language models (LLMs) has primarily focused on model parameters and training data size, overlooking the role of vocabulary size. We investigate how vocabulary size impacts LLM scaling laws by training models ranging from 33M to 3B parameters on up to 500B characters with various vocabulary configurations. We propose three complementary approaches for predicting the compute-optimal vocabulary size: IsoFLOPs analysis, derivative estimation, and parametric fit of the loss function. Our approaches converge on the conclusion that the optimal vocabulary size depends on the compute budget, with larger models requiring larger vocabularies. Most LLMs, however, use insufficient vocabulary sizes. For example, we predict that the optimal vocabulary size of Llama2-70B should have been at least 216K, 7 times larger than its vocabulary of 32K. We validate our predictions empirically by training models with 3B parameters across different FLOPs budgets. Adopting our predicted optimal vocabulary size consistently improves downstream performance over commonly used vocabulary sizes. By increasing the vocabulary size from the conventional 32K to 43K, we improve performance on ARC-Challenge from 29.1 to 32.0 with the same 2.3e21 FLOPs. Our work highlights the importance of jointly considering tokenization and model scaling for efficient pre-training.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2407.13623")
get_code_for_paper("2407.13623")
have("2407.13623")
Connect an agent — have() is free.