We lifted 14 functions out of this paper's own repositories and ran 7 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| ablustrund/loramoe | canonical | 7 of 14 |
| Function | Status | Where it lives |
|---|---|---|
| convert_pytorch_state_dict_to_flax | Ran | ablustrund/loramoe/transformers/modeling_flax_pytorch_utils.py pointer only (licence: NONE) · get_code("5168a172bbb363a5") |
| get_imports | Ran | ablustrund/loramoe/transformers/dynamic_module_utils.py pointer only (licence: NONE) · get_code("ccaa0edba373622c") |
| infer_metric_tags_from_eval_results | Ran | ablustrund/loramoe/transformers/modelcard.py pointer only (licence: NONE) · get_code("dd8f7751ef859581") |
| parse_keras_history | Ran | ablustrund/loramoe/transformers/modelcard.py pointer only (licence: NONE) · get_code("a4491b45d19b7a39") |
| prepare_model_for_kbit_training | Ran | ablustrund/loramoe/run_loramoe.py pointer only (licence: NONE) · get_code("591cff6b55ba6bfb") |
| quick_gelu | Ran | ablustrund/loramoe/transformers/modeling_flax_utils.py pointer only (licence: NONE) · get_code("a66d2b78670c85e1") |
| rename_key_and_reshape_tensor | Ran | ablustrund/loramoe/transformers/modeling_flax_pytorch_utils.py pointer only (licence: NONE) · get_code("61779b0406a59cfc") |
| build_instruction_dataset | Not yet run | ablustrund/loramoe/build_dataset.py pointer only (licence: NONE) · get_code("2776dfe4cf53097d") |
| dtype_byte_size | Not yet run | ablustrund/loramoe/transformers/modeling_flax_utils.py pointer only (licence: NONE) · get_code("9d98806acb3144b7") |
| flax_shard_checkpoint | Not yet run | ablustrund/loramoe/transformers/modeling_flax_utils.py pointer only (licence: NONE) · get_code("e3ce70ca9258800e") |
| forward | Not yet run | ablustrund/loramoe/flash_attn_patch.py pointer only (licence: NONE) · get_code("602b6b6c04496cfd") |
| get_relative_import_files | Not yet run | ablustrund/loramoe/transformers/dynamic_module_utils.py pointer only (licence: NONE) · get_code("00cc853a73f3aeb6") |
| get_relative_imports | Not yet run | ablustrund/loramoe/transformers/dynamic_module_utils.py pointer only (licence: NONE) · get_code("450aeffe7f80f917") |
| is_hf_dataset | Not yet run | ablustrund/loramoe/transformers/modelcard.py pointer only (licence: NONE) · get_code("ec7b1528a936234d") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Supervised fine-tuning (SFT) is a crucial step for large language models (LLMs), enabling them to align with human instructions and enhance their capabilities in downstream tasks. Increasing instruction data substantially is a direct solution to align the model with a broader range of downstream tasks or notably improve its performance on a specific task. However, we find that large-scale increases in instruction data can damage the world knowledge previously stored in LLMs. To address this challenge, we propose LoRAMoE, a novelty framework that introduces several low-rank adapters (LoRA) and integrates them by using a router network, like a plugin version of Mixture of Experts (MoE). It freezes the backbone model and forces a portion of LoRAs to focus on leveraging world knowledge to solve downstream tasks, to alleviate world knowledge-edge forgetting. Experimental results show that, as the instruction data increases, LoRAMoE can significantly improve the ability to process downstream tasks, while maintaining the world knowledge stored in the LLM.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2312.09979")
get_code_for_paper("2312.09979")
have("2312.09979")
Connect an agent — have() is free.