We lifted 19 functions out of this paper's own repositories and ran 14 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| lcs2-iiitd/monteclora | canonical | 11 of 16 |
| copy not recorded | — | 3 of 3 |
| Function | Status | Where it lives |
|---|---|---|
| b2mb | Ran | this paper's copy was not recorded; identical code first harvested from huggingface/peft pointer only · get_code("32380a050ca033d4") |
| build_tip_message | Ran | lcs2-iiitd/monteclora/transformers/utils/deprecate_models.py code served (permissive licence) · get_code("ab306165fa74656c") |
| generate_prompt | Ran | lcs2-iiitd/monteclora/finetune.py code served (permissive licence) · get_code("56e674752db00649") |
| generate_prompt | Ran | lcs2-iiitd/monteclora/generate.py code served (permissive licence) · get_code("0dbc7fc6c518d0fb") |
| generate_prompt | Ran | lcs2-iiitd/monteclora/commonsense_evaluate.py code served (permissive licence) · get_code("1c6b8be890b6118a") |
| get_closest_label | Ran | this paper's copy was not recorded; identical code first harvested from huggingface/peft pointer only · get_code("bdad3d08a6241ede") |
| get_config_class_from_processor_class | Ran | lcs2-iiitd/monteclora/transformers/utils/create_dummy_models.py code served (permissive licence) · get_code("3d999b95009c7537") |
| get_list_of_repo_model_paths | Ran | lcs2-iiitd/monteclora/transformers/utils/models_to_deprecate.py code served (permissive licence) · get_code("938fc7c2422ef91d") |
| get_model_doc_path | Ran | lcs2-iiitd/monteclora/transformers/utils/deprecate_models.py code served (permissive licence) · get_code("051d790d9b197d9d") |
| get_processor_types_from_config_class | Ran | lcs2-iiitd/monteclora/transformers/utils/create_dummy_models.py code served (permissive licence) · get_code("0c0b81b081919d8e") |
| levenshtein_distance | Ran | this paper's copy was not recorded; identical code first harvested from huggingface/peft pointer only · get_code("d267d33e45b87bcd") |
| load_data | Ran | lcs2-iiitd/monteclora/commonsense_evaluate.py code served (permissive licence) · get_code("56a9466ff35b35bb") |
| replace_args_in_script | Ran | lcs2-iiitd/monteclora/expt.py code served (permissive licence) · get_code("f440299af78ed374") |
| write_temp_script | Ran | lcs2-iiitd/monteclora/expt.py code served (permissive licence) · get_code("0667181f616b34bb") |
| extract_model_info | Not yet run | lcs2-iiitd/monteclora/transformers/utils/deprecate_models.py code served (permissive licence) · get_code("54b7273383fcfc2d") |
| get_architectures_from_config_class | Not yet run | lcs2-iiitd/monteclora/transformers/utils/create_dummy_models.py code served (permissive licence) · get_code("b176efb26f72e507") |
| main | Not yet run | lcs2-iiitd/monteclora/commonsense_evaluate.py code served (permissive licence) · get_code("93ae876f7bcc5547") |
| main | Not yet run | lcs2-iiitd/monteclora/evaluate.py code served (permissive licence) · get_code("2982bc8c318ebd42") |
| train | Not yet run | lcs2-iiitd/monteclora/finetune.py code served (permissive licence) · get_code("7852c586e42377d2") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Large Language Models (LLMs) are highly resource-intensive to fine-tune due to their enormous size. While low-rank adaptation is a prominent parameter-efficient fine-tuning approach, it suffers from sensitivity to hyperparameter choices, leading to instability in model performance on fine-tuning downstream tasks. This paper highlights the importance of effective parameterization in low-rank fine-tuning to reduce estimator variance and enhance the stability of final model outputs. We propose MonteCLoRA, an efficient fine-tuning technique that employs Monte Carlo estimation to learn an unbiased posterior estimation of low-rank parameters with low expected variance, stabilizing fine-tuned LLMs with only O(r) additional parameters, for a given rank r. MonteCLoRA shows 0.5% and 1.6% improvements in accuracy and robustness over unregularized low-rank adaptation method on natural language understanding tasks with pre-trained RoBERTa-base. Furthermore, in generative tasks with pre-trained LLaMA-1-7B and LLaMA-3.2-3B-Instruct, MonteCLoRA demonstrates robust performance with 50% and 62% lower spreads respectively than the contemporary efficient fine-tuning methods. The theoretical and empirical results presented in the paper underscore how parameterization and hyperpriors balance exploration-exploitation in the low-rank parametric space, therefore leading to more optimal and robust parameter estimation during efficient fine-tuning.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2411.04358")
get_code_for_paper("2411.04358")
have("2411.04358")
Connect an agent — have() is free.