We lifted 3 functions out of this paper's own repositories and ran 3 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| joeljang/elm | canonical | 2 of 2 |
| copy not recorded | — | 1 of 1 |
| Function | Status | Where it lives |
|---|---|---|
| clease_jinja | Ran | joeljang/elm/retrieval/run_retrieval.py pointer only (licence: NONE) · get_code("9aa2aaf773117b75") |
| euclidean_distance | Ran | joeljang/elm/retrieval/run_retrieval.py pointer only (licence: NONE) · get_code("d77e7ade95e0c350") |
| run_command | Ran | this paper's copy was not recorded; identical code first harvested from rabeehk/compacter pointer only · get_code("44e48dfbdecee3da") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Recently, Language Models (LMs) instruction-tuned on multiple tasks, also known as multitask-prompted fine-tuning (MT), have shown the capability to generalize to unseen tasks. Previous work has shown that scaling the number of training tasks is the key component in making stronger MT LMs. In this work, we report an unexpected finding that an expert LM fine-tuned on just a single task can outperform an MT LM trained with 300+ different tasks on 11 different unseen datasets and on 13 datasets of the BIG-bench benchmark by a mean accuracy of 3.20% and 1.29%, respectively. This finding casts doubt on the previously held belief that simply scaling the number of tasks makes stronger MT LMs. Leveraging this finding, we further show that this distributed approach of training a separate expert LM per training task instead of a single MT LM for zero-shot inference possesses many benefits including (1) avoiding negative task transfer that often occurs during instruction tuning, (2) being able to continually learn new tasks without having to re-train on previous tasks to avoid catastrophic forgetting, and (3) showing compositional capabilities when merging individual experts together. The code is available at https://github.com/joeljang/ELM.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2302.03202")
get_code_for_paper("2302.03202")
have("2302.03202")
Connect an agent — have() is free.