Mohammad Khan, Daniel Cremers, Thomas Möllenhoff, Yuesong Shen, Nico Daheim, Bai Cong, Rio Yokota
We lifted 4 functions out of this paper's own repositories and ran 0 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| team-approx-bayes/ivon-lora | canonical | 0 of 4 |
| Function | Status | Where it lives |
|---|---|---|
| calculate_metrics | Not yet run | team-approx-bayes/ivon-lora/utils.py pointer only (licence: NONE) · get_code("436f4c309d8668cf") |
| get_model_with_lora | Not yet run | team-approx-bayes/ivon-lora/model.py pointer only (licence: NONE) · get_code("409ef49365463547") |
| get_processed_datasets | Not yet run | team-approx-bayes/ivon-lora/data.py pointer only (licence: NONE) · get_code("ebd2a7cc208c71af") |
| get_raw_datasets | Not yet run | team-approx-bayes/ivon-lora/data.py pointer only (licence: NONE) · get_code("d8835e01bd280f77") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
We show that variational learning can significantly improve the accuracy and calibration of Low-Rank Adaptation (LoRA) without a substantial increase in the cost. We replace AdamW by the Improved Variational Online Newton (IVON) algorithm to finetune large language models. For Llama-2 with 7 billion parameters, IVON improves the accuracy over AdamW by 2.8% and expected calibration error by 4.6%. The accuracy is also better than the other Bayesian alternatives, yet the cost is lower and the implementation is easier. Our work provides additional evidence for the effectiveness of IVON for large language models. The code is available at https://github.com/team-approx-bayes/ivon-lora.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2411.04421")
get_code_for_paper("2411.04421")
have("2411.04421")
Connect an agent — have() is free.