We lifted 4 functions out of this paper's own repositories and ran 0 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| nikhil-ghosh-berkeley/loraplus | canonical | 0 of 4 |
| Function | Status | Where it lives |
|---|---|---|
| attempt_train | Not yet run | nikhil-ghosh-berkeley/loraplus/glue/src/train_utils.py code served (permissive licence) · get_code("70ab4a45d0c68e3e") |
| create_loraplus_optimizer | Not yet run | nikhil-ghosh-berkeley/loraplus/lora_plus.py code served (permissive licence) · get_code("e5ef69b1e4b7a75a") |
| find_valid_checkpoint | Not yet run | nikhil-ghosh-berkeley/loraplus/glue/src/train_utils.py code served (permissive licence) · get_code("00c123899f8febfb") |
| get_module | Not yet run | nikhil-ghosh-berkeley/loraplus/lora_plus.py code served (permissive licence) · get_code("ef1ba16faaacbb49") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
In this paper, we show that Low Rank Adaptation (LoRA) as originally introduced in Hu et al. (2021) leads to suboptimal finetuning of models with large width (embedding dimension). This is due to the fact that adapter matrices A and B in LoRA are updated with the same learning rate. Using scaling arguments for large width networks, we demonstrate that using the same learning rate for A and B does not allow efficient feature learning. We then show that this suboptimality of LoRA can be corrected simply by setting different learning rates for the LoRA adapter matrices A and B with a well-chosen ratio. We call this proposed algorithm LoRA$+$. In our extensive experiments, LoRA$+$ improves performance (1-2 $\%$ improvements) and finetuning speed (up to $\sim$ 2X SpeedUp), at the same computational cost as LoRA.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2402.12354")
get_code_for_paper("2402.12354")
have("2402.12354")
Connect an agent — have() is free.