Oyvind Tafjord, Kyle Lo, Iz Beltagy, Jesse Dodge, Kyle Richardson, Valentin Hofmann, Yanai Elazar, Akshita Bhagia, Ian Magnusson, Abhilasha Ravichander, Dustin Schwenk, Pete Walsh, and 24 more
We lifted 9 functions out of this paper's own repositories and ran 7 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| allenai/dolma | canonical | 7 of 9 |
| Function | Status | Where it lives |
|---|---|---|
| cumsum_with_reset | Ran | allenai/dolma/python/dolma/core/binning.py code served (permissive licence) · get_code("c303e2effe4bbce7") |
| equal_count_hist | Ran | allenai/dolma/python/dolma/core/binning.py code served (permissive licence) · get_code("0acb026cf0d9a188") |
| get_logger | Ran | allenai/dolma/python/dolma/core/loggers.py code served (permissive licence) · get_code("13c0d6acf12fe992") |
| is_local | Ran | allenai/dolma/python/dolma/core/paths.py code served (permissive licence) · get_code("f68e6fea546f6b07") |
| remove_params | Ran | allenai/dolma/python/dolma/core/paths.py code served (permissive licence) · get_code("c5eb06dd7524b7df") |
| round_values_for_visual | Ran | allenai/dolma/python/dolma/core/analyzer.py code served (permissive licence) · get_code("7469f54cf29978e9") |
| sort_and_merge_bins | Ran | allenai/dolma/python/dolma/core/binning.py code served (permissive licence) · get_code("5afe4332ff5811b5") |
| delete_file | Not yet run | allenai/dolma/python/dolma/core/paths.py code served (permissive licence) · get_code("703dcf9d8b374274") |
| gzip_open | Not yet run | allenai/dolma/python/dolma/core/ft_dataset.py code served (permissive licence) · get_code("278c8e78687fe425") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Information about pretraining corpora used to train the current best-performing language models is seldom discussed: commercial models rarely detail their data, and even open models are often released without accompanying training data or recipes to reproduce them. As a result, it is challenging to conduct and advance scientific research on language modeling, such as understanding how training data impacts model capabilities and limitations. To facilitate scientific research on language model pretraining, we curate and release Dolma, a three-trillion-token English corpus, built from a diverse mixture of web content, scientific papers, code, public-domain books, social media, and encyclopedic materials. We extensively document Dolma, including its design principles, details about its construction, and a summary of its contents. We present analyses and experimental results on intermediate states of Dolma to share what we have learned about important data curation practices. Finally, we open-source our data curation toolkit to enable reproduction of our work as well as support further research in large-scale data curation.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2402.00159")
get_code_for_paper("2402.00159")
have("2402.00159")
Connect an agent — have() is free.