We lifted 3 functions out of this paper's own repositories and ran 3 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| mohammadrezabanaei/lora-xs | canonical | 3 of 3 |
| Function | Status | Where it lives |
|---|---|---|
| get_replacement_module | Ran | mohammadrezabanaei/lora-xs/utils/initialization_utils.py pointer only (licence: NONE) · get_code("4e7063af2052f801") |
| kaiming_uniform_init | Ran | mohammadrezabanaei/lora-xs/utils/initialization_utils.py pointer only (licence: NONE) · get_code("6102fbd31542a51f") |
| kaiming_uniform_init_lower_half | Ran | mohammadrezabanaei/lora-xs/utils/initialization_utils.py pointer only (licence: NONE) · get_code("d24ba8b4d81718d9") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
The growth of large language models underscores the need for parameter-efficient fine-tuning. Despite its popularity, LoRA encounters storage and computational challenges when deploying multiple task- or user-specific modules. To address this, we introduce LoRA-XS, a novel fine-tuning method backed by a theoretical derivation. LoRA-XS drastically reduces trainable parameters by incorporating a small, trainable weight matrix between frozen low-rank matrices derived from the Singular Value Decomposition of pre-trained weights. This design enables LoRA-XS to reduce storage requirements by over 100x in 7B models compared to LoRA. Additionally, unlike other methods, LoRA-XS imposes no lower bound on trainable parameters - it can scale from a single parameter per module to arbitrarily large values, adapting to any storage or computational constraint. Evaluations on GLUE, GSM8K, MATH, and commonsense reasoning benchmarks across different model scales reveal that LoRA-XS consistently outperforms or matches LoRA and VeRA in accuracy, offering unmatched parameter efficiency. Our ablation studies highlight the significance of singular vectors in transformer weights, establishing LoRA-XS as a powerful, storage-efficient solution for scaling and personalizing large language models.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2405.17604")
get_code_for_paper("2405.17604")
have("2405.17604")
Connect an agent — have() is free.