We lifted 12 functions out of this paper's own repositories and ran 11 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| sehyunkwon/task-diversity-icl | canonical | 11 of 12 |
| Function | Status | Where it lives |
|---|---|---|
| accuracy | Ran | sehyunkwon/task-diversity-icl/src/bool_task.py pointer only (licence: NONE) · get_code("d7ae3e05768f422d") |
| adjust_batch_sizes | Ran | sehyunkwon/task-diversity-icl/task_configs/generate.py pointer only (licence: NONE) · get_code("420674d96c9c88ae") |
| cross_entropy | Ran | sehyunkwon/task-diversity-icl/src/bool_task.py pointer only (licence: NONE) · get_code("41065e1207ba1b0d") |
| cross_entropy | Ran | sehyunkwon/task-diversity-icl/src/continuous_task.py pointer only (licence: NONE) · get_code("797281bf91c00aef") |
| generate_idx_list | Ran | sehyunkwon/task-diversity-icl/src/bool_task.py pointer only (licence: NONE) · get_code("1ee7db3dafc63aa0") |
| get_optimal_ncl_acc | Ran | sehyunkwon/task-diversity-icl/src/ncl.py pointer only (licence: NONE) · get_code("71292b149b12d917") |
| get_optimal_ncl_loss | Ran | sehyunkwon/task-diversity-icl/src/ncl.py pointer only (licence: NONE) · get_code("0157552fb1f390d1") |
| mean_squared_error | Ran | sehyunkwon/task-diversity-icl/src/continuous_task.py pointer only (licence: NONE) · get_code("fc94f27a0973e093") |
| nn_scoring_step | Ran | sehyunkwon/task-diversity-icl/src/attention_analysis.py pointer only (licence: NONE) · get_code("a81285c7684656ca") |
| prefix_scoring_step | Ran | sehyunkwon/task-diversity-icl/src/attention_analysis.py pointer only (licence: NONE) · get_code("211a2ef589b849d4") |
| squared_error | Ran | sehyunkwon/task-diversity-icl/src/continuous_task.py pointer only (licence: NONE) · get_code("348d63a78eb4cb94") |
| get_final_var | Not yet run | sehyunkwon/task-diversity-icl/src/curriculum.py pointer only (licence: NONE) · get_code("3c24ac252fef0868") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
In-context learning (ICL) describes a language model's ability to generate outputs based on a set of input demonstrations and a subsequent query. To understand this remarkable capability, researchers have studied simplified, stylized models. These studies have consistently observed long loss plateaus, during which models exhibit minimal improvement, followed by a sudden, rapid surge of learning. In this work, we reveal that training on multiple diverse ICL tasks simultaneously shortens the loss plateaus, making each task easier to learn. This finding is surprising as it contradicts the natural intuition that the combined complexity of multiple ICL tasks would lengthen the learning process, not shorten it. Our result suggests that the recent success in large-scale training of language models may be attributed not only to the richness of the data at scale but also to the easier optimization (training) induced by the diversity of natural language training data.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2410.05448")
get_code_for_paper("2410.05448")
have("2410.05448")
Connect an agent — have() is free.