We lifted 3 functions out of this paper's own repositories and ran 3 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| catherinearnett/byte-premium-tool | canonical | 3 of 3 |
| Function | Status | Where it lives |
|---|---|---|
| calculate_pairwise_premium_from_parallel | Ran | catherinearnett/byte-premium-tool/byte_premium_tool.py code served (permissive licence) · get_code("9c715bba08d58762") |
| get_premium_from_regression | Ran | catherinearnett/byte-premium-tool/byte_premium_tool.py code served (permissive licence) · get_code("43302aa62fe5a8d6") |
| retrieve_calculated_premium | Ran | catherinearnett/byte-premium-tool/byte_premium_tool.py code served (permissive licence) · get_code("5e300d5435c0cc5c") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
How should text dataset sizes be compared across languages? Even for content-matched (parallel) corpora, UTF-8 encoded text can require a dramatically different number of bytes for different languages. In our work, we define the byte premium between two languages as the ratio of bytes used to encode content-matched text in those languages. We compute byte premiums for 1155 languages, and we use linear regressions to estimate byte premiums for other languages. We release a tool to obtain byte premiums for any two languages, enabling comparisons of dataset sizes across languages for more equitable multilingual model development and data practices.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2403.00686")
get_code_for_paper("2403.00686")
have("2403.00686")
Connect an agent — have() is free.