We lifted 10 functions out of this paper's own repositories and ran 9 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| yuvalkirstain/lm-evaluation-harness | canonical | 9 of 10 |
| Function | Status | Where it lives |
|---|---|---|
| mean | Ran | yuvalkirstain/lm-evaluation-harness/lm_eval/metrics.py code served (permissive licence) · get_code("0928f497e20fb443") |
| get_result | Ran | yuvalkirstain/lm-evaluation-harness/lm_eval/models/gpt3.py code served (permissive licence) · get_code("65ed12119df56f3b") |
| group | Ran | yuvalkirstain/lm-evaluation-harness/lm_eval/utils.py code served (permissive licence) · get_code("359c35fdcd7a0ee5") |
| hash_args | Ran | yuvalkirstain/lm-evaluation-harness/lm_eval/base.py code served (permissive licence) · get_code("ea06eaae4fc1eaf0") |
| is_number | Ran | yuvalkirstain/lm-evaluation-harness/lm_eval/utils.py code served (permissive licence) · get_code("e18607dae38b30e2") |
| matthews_corrcoef | Ran | yuvalkirstain/lm-evaluation-harness/lm_eval/metrics.py code served (permissive licence) · get_code("028eb6dad1493f40") |
| median | Ran | yuvalkirstain/lm-evaluation-harness/lm_eval/metrics.py code served (permissive licence) · get_code("79a89132ce3f75f1") |
| simple_parse_args_string | Ran | yuvalkirstain/lm-evaluation-harness/lm_eval/utils.py code served (permissive licence) · get_code("c6d952282e302ade") |
| write_datasets_fo_read | Ran | yuvalkirstain/lm-evaluation-harness/lm_eval/train_pl.py code served (permissive licence) · get_code("31b87866d085212b") |
| load_raw_datasests | Not yet run | yuvalkirstain/lm-evaluation-harness/lm_eval/train_pl.py code served (permissive licence) · get_code("3b0f3f81457fce8a") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
We investigate the dynamics of increasing the number of model parameters versus the number of labeled examples across a wide variety of tasks. Our exploration reveals that while scaling parameters consistently yields performance improvements, the contribution of additional examples highly depends on the task's format. Specifically, in open question answering tasks, enlarging the training set does not improve performance. In contrast, classification, extractive question answering, and multiple choice tasks benefit so much from additional examples that collecting a few hundred examples is often "worth" billions of parameters. We hypothesize that unlike open question answering, which involves recalling specific information, solving strategies for tasks with a more restricted output space transfer across examples, and can therefore be learned with small amounts of labeled data.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2110.04374")
get_code_for_paper("2110.04374")
have("2110.04374")
Connect an agent — have() is free.