Label-Free Cross-Task LoRA Merging with Null-Space Compression

Kaist, Kuk-Jin Yoon, Wooseong Jeong, Wonyoung Lee

arXiv · PDF · Open in the Atlas

Code that ran

We lifted 2 functions out of this paper's own repositories and ran 1 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.

Repositories linked to this paper

• wonyoung01/nsc_merging

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Model merging combines independently fine-tuned checkpoints without joint multi-task training. In the era of foundation-model, fine-tuning with Low-Rank Adaptation (LoRA) is prevalent, making LoRA merging a promising target. Existing approaches can work in homogeneous settings where all target tasks are classification but often fail when tasks span classification and regression. Approaches using entropy-based surrogates do not apply to regression and are costly for large language models due to long token sequences. We introduce Null-Space Compression (NSC) Merging, a label-free, output-agnostic method that sets merge weights from adapter geometry. Our key observation is that during LoRA finetuning the down-projection factor A in ∆W = BA compresses its null space, and the compression correlates with performance. NSC uses this as an optimization signal for merging that can generalize across classification, regression, and sequence generation. NSC achieves state-of-the-art performance across twenty heterogeneous vision tasks with balanced gains where prior methods overfit subsets of tasks. It also outperforms baselines on six NLI benchmarks and on vision-language evaluations for VQA and image captioning, demonstrating scalability and effectiveness. Our code is available at https: //github.com/wonyoung01/nsc_merging

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2603.26317")
get_code_for_paper("2603.26317")
have("2603.26317")

Connect an agent — have() is free.