Zheng Zhang, Yifan Yang, Ngai Wong, Jiajun Zhou
We lifted 8 functions out of this paper's own repositories and ran 4 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| yifanycc/loretta | canonical | 4 of 8 |
| Function | Status | Where it lives |
|---|---|---|
| build_message | Ran | yifanycc/loretta/large_models/message.py pointer only (licence: GPL-3.0) · get_code("665af05400a43244") |
| find_module | Ran | yifanycc/loretta/large_models/lora.py pointer only (licence: GPL-3.0) · get_code("b5adfd6ac99a1b18") |
| get_parameter_number | Ran | yifanycc/loretta/bert_model/run_glue_v5.py pointer only (licence: GPL-3.0) · get_code("918fc46b5088d174") |
| normalize_answer | Ran | yifanycc/loretta/bert_model/metrics.py pointer only (licence: GPL-3.0) · get_code("e7e75981cb464788") |
| attn_forward_hook | Not yet run | yifanycc/loretta/large_models/prefix.py pointer only (licence: GPL-3.0) · get_code("bdb460cb247501dc") |
| calculate_metric | Not yet run | yifanycc/loretta/bert_model/metrics.py pointer only (licence: GPL-3.0) · get_code("84c658f2c9252878") |
| f1 | Not yet run | yifanycc/loretta/bert_model/metrics.py pointer only (licence: GPL-3.0) · get_code("b989d4bce26f77ce") |
| prepare_inputs_for_generation | Not yet run | yifanycc/loretta/large_models/prefix.py pointer only (licence: GPL-3.0) · get_code("84920a8095dca4fa") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Various parameter-efficient fine-tuning (PEFT) techniques have been proposed to enable computationally efficient fine-tuning while maintaining model performance. However, existing PEFT methods are still limited by the growing number of trainable parameters with the rapid deployment of Large Language Models (LLMs). To address this challenge, we present LoRETTA, an ultra-parameter-efficient framework that significantly reduces trainable parameters through tensor-train decomposition. Specifically, we propose two methods, named LoRETTA adp and LoRETTA rep . The former employs tensorized adapters, offering a high-performance yet lightweight approach for the fine-tuning of LLMs. The latter emphasizes fine-tuning via weight reparameterization with a set of small tensor factors. LoRETTA achieves comparable or better performance than most widely used PEFT methods with up to 100× fewer parameters on the LLaMA-2-7B models. Furthermore, empirical results demonstrate that the proposed methods exhibit remarkable anti-overfitting capability, effectively improve training efficiency, and enjoy better multi-task learning performance. Plug-andplay loretta library built upon the Huggingface framework and PEFT library are provided. ‡
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2402.11417")
get_code_for_paper("2402.11417")
have("2402.11417")
Connect an agent — have() is free.