We lifted 8 functions out of this paper's own repositories and ran 6 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| cerebras/modelzoo | canonical | 6 of 8 |
| Function | Status | Where it lives |
|---|---|---|
| add_run_args | Ran | cerebras/modelzoo/src/cerebras/modelzoo/cli/utils.py code served (permissive licence) · get_code("eb763a431940c30f") |
| diff_checkpoints | Ran | cerebras/modelzoo/src/cerebras/modelzoo/cli/checkpoint_cli.py code served (permissive licence) · get_code("9afc884551e7d9f0") |
| flatten_optimizer_params | Ran | cerebras/modelzoo/src/cerebras/modelzoo/common/optim_utils.py code served (permissive licence) · get_code("7d11773480563b1d") |
| is_dir | Ran | cerebras/modelzoo/src/cerebras/modelzoo/cli/utils.py code served (permissive licence) · get_code("6e0d1515d25fe6d0") |
| to_cpu | Ran | cerebras/modelzoo/src/cerebras/modelzoo/common/pytorch_utils.py code served (permissive licence) · get_code("6e21827728420a8b") |
| to_tensor | Ran | cerebras/modelzoo/src/cerebras/modelzoo/common/pytorch_utils.py code served (permissive licence) · get_code("27f84a5a2a5d8e00") |
| copy_config | Not yet run | cerebras/modelzoo/src/cerebras/modelzoo/cli/utils.py code served (permissive licence) · get_code("eef378d38e298a9f") |
| diff_checkpoints_from_file | Not yet run | cerebras/modelzoo/src/cerebras/modelzoo/cli/checkpoint_cli.py code served (permissive licence) · get_code("28232adaf791ee88") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
We introduce the Bittensor Language Model, called "BTLM-3B-8K", a new state-of-the-art 3 billion parameter open-source language model. BTLM-3B-8K was trained on 627B tokens from the SlimPajama dataset with a mixture of 2,048 and 8,192 context lengths. BTLM-3B-8K outperforms all existing 3B parameter models by 2-5.5% across downstream tasks. BTLM-3B-8K is even competitive with some 7B parameter models. Additionally, BTLM-3B-8K provides excellent long context performance, outperforming MPT-7B-8K and XGen-7B-8K on tasks up to 8,192 context length. We trained the model on a cleaned and deduplicated SlimPajama dataset; aggressively tuned the \textmu P hyperparameters and schedule; used ALiBi position embeddings; and adopted the SwiGLU nonlinearity. On Hugging Face, the most popular models have 7B parameters, indicating that users prefer the quality-size ratio of 7B models. Compacting the 7B parameter model to one with 3B parameters, with little performance impact, is an important milestone. BTLM-3B-8K needs only 3GB of memory with 4-bit precision and takes 2.5x less inference compute than 7B models, helping to open up access to a powerful language model on mobile and edge devices. BTLM-3B-8K is available under an Apache 2.0 license on Hugging Face: https://huggingface.co/cerebras/btlm-3b-8k-base.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2309.11568")
get_code_for_paper("2309.11568")
have("2309.11568")
Connect an agent — have() is free.