We lifted 6 functions out of this paper's own repositories and ran 4 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| gstoica27/knots | canonical | 4 of 6 |
| Function | Status | Where it lives |
|---|---|---|
| resolve_sign | Ran | gstoica27/knots/masking_ops.py code served (permissive licence) · get_code("70e49dc222d0391a") |
| resolve_zero_signs | Ran | gstoica27/knots/masking_ops.py code served (permissive licence) · get_code("860408c2bd01950e") |
| topk_values_mask | Ran | gstoica27/knots/merging_functions.py code served (permissive licence) · get_code("f6ead996d6b1adba") |
| topk_values_mask | Ran | gstoica27/knots/masking_ops.py code served (permissive licence) · get_code("cc25fa48e243d80b") |
| get_model_from_config | Not yet run | gstoica27/knots/models/huggingface_clip.py code served (permissive licence) · get_code("1937c8308a2b4783") |
| tv_merging | Not yet run | gstoica27/knots/merging_functions.py code served (permissive licence) · get_code("3521eec6c7977199") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Recent model merging methods demonstrate that the parameters of fully-finetuned models specializing in distinct tasks can be combined into one model capable of solving all tasks without retraining. Yet, this success does not transfer well when merging LoRA finetuned models. We study this phenomenon and observe that the weights of LoRA finetuned models showcase a lower degree of alignment compared to their fully-finetuned counterparts. We hypothesize that improving this alignment is key to obtaining better LoRA model merges, and propose KnOTS to address this problem. KnOTS uses the SVD to jointly transform the weights of different LoRA models into an aligned space, where existing merging methods can be applied. In addition, we introduce a new benchmark that explicitly evaluates whether merged models are general models. Notably, KnOTS consistently improves LoRA merging by up to 4.3% across several vision and language benchmarks, including our new setting. We release our code at: https://github.com/gstoica27/KnOTS.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2410.19735")
get_code_for_paper("2410.19735")
have("2410.19735")
Connect an agent — have() is free.