Hannaneh Hajishirzi, Noah Smith, Yizhong Wang, Iz Beltagy, Hamish Ivison, ♣ Pradeep, Dasigi Jack, Hessel Tushar, Khot Khyathi, Raghavi Chandu, David Wadden, Kelsey Macmillan
We lifted 12 functions out of this paper's own repositories and ran 6 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| allenai/open-instruct | canonical | 5 of 8 |
| schwartz-lab-NLP/label-bias | — | 1 of 2 |
| john-hewitt/implicit-ins | — | 0 of 1 |
| Forence1999/PRoLoRA | — | 0 of 1 |
| Function | Status | Where it lives |
|---|---|---|
| entropy_from_logits | Ran | allenai/open-instruct/open_instruct/model_utils.py code served (permissive licence) · get_code("3ca03473e153b197") |
| AssistantSpanDerivationError | Ran | allenai/open-instruct/open_instruct/dataset_transformation.py code served (permissive licence) · get_code("7c7fbc84c32888c3") |
| _assistant_token_spans_from_prefix_lengths | Ran | allenai/open-instruct/open_instruct/dataset_transformation.py code served (permissive licence) · get_code("d38e226c6b168c4a") |
| _trainable_assistant_indices | Ran | allenai/open-instruct/open_instruct/dataset_transformation.py code served (permissive licence) · get_code("4baec48bd245f848") |
| _was_truncated | Ran | allenai/open-instruct/open_instruct/dataset_transformation.py code served (permissive licence) · get_code("29d4865962de784a") |
| find_all_linear_names | Ran | schwartz-lab-NLP/label-bias/src/superni/utils/lora_utils.py pointer only (licence: NONE) · get_code("bea710de6ec7c18d") |
| CombinedCausalLM | Not yet run | john-hewitt/implicit-ins/open_instruct/combined_model.py code served (permissive licence) · get_code("a7124b520aa932e3") |
| LoraLayer_update_layer | Not yet run | Forence1999/PRoLoRA/modules/chunkwise_sharing_LoRA.py code served (permissive licence) · get_code("370f948ce836365d") |
| _tokenize_tulu_sft_with_assistant_labels | Not yet run | allenai/open-instruct/open_instruct/dataset_transformation.py code served (permissive licence) · get_code("b027fd7ccc6d4be9") |
| _verify_assistant_spans_cover_content | Not yet run | allenai/open-instruct/open_instruct/dataset_transformation.py code served (permissive licence) · get_code("cfe06104ea9d81d9") |
| prepare_peft_model | Not yet run | schwartz-lab-NLP/label-bias/src/superni/utils/lora_utils.py pointer only (licence: NONE) · get_code("d5011eb064a2523d") |
| take_and_check_valid_input | Not yet run | allenai/open-instruct/scripts/add_metadata.py code served (permissive licence) · get_code("b252d3f8f3f983ba") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
In this work we explore recent advances in instruction-tuning language models on a range of open instruction-following datasets. Despite recent claims that open models can be on par with state-of-the-art proprietary models, these claims are often accompanied by limited evaluation, making it difficult to compare models across the board and determine the utility of various resources. We provide a large set of instruction-tuned models from 6.7B to 65B parameters in size, trained on 12 instruction datasets ranging from manually curated (e.g., OpenAssistant) to synthetic and distilled (e.g., Alpaca) and systematically evaluate them on their factual knowledge, reasoning, multilinguality, coding, safety, and open-ended instruction following abilities through a collection of automatic, model-based, and human-based metrics. We further introduce TÜLU , our best performing instruction-tuned model suite finetuned on a combination of high-quality open resources. Our experiments show that different instruction-tuning datasets can uncover or enhance specific skills, while no single dataset (or combination) provides the best performance across all evaluations. Interestingly, we find that model and human preference-based evaluations fail to reflect differences in model capabilities exposed by benchmark-based evaluations, suggesting the need for the type of systemic evaluation performed in this work. Our evaluations show that the best model in any given evaluation reaches on average 87% of ChatGPT performance, and 73% of GPT-4 performance, suggesting that further investment in building better base models and instruction-tuning data is required to close the gap. We release our instruction-tuned models, including a fully finetuned 65B TÜLU , along with our code, data, and evaluation framework to facilitate future research. 2 * Equal contribution.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2306.04751")
get_code_for_paper("2306.04751")
have("2306.04751")
Connect an agent — have() is free.