Gradient-Sign Masking for Task Vector Transport Across Pre-Trained Models

Angelo Porrello, Simone Calderara, Marco Ciccone, Fengyuan Liu, Aniello Panariello, Filippo Rinaldi, Giacomo Salici

arXiv · PDF · Open in the Atlas

Code that ran

We lifted 2 functions out of this paper's own repositories and ran 1 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.

Repositories linked to this paper

• fillo-rinaldi/GradFix

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

When a new release of a foundation model is published, practitioners typically need to repeat fine-tuning, even if the same task was already tackled in the previous version. A promising alternative is to reuse the parameter changes (i.e., task vectors) that capture how a model adapts to a specific task. However, these vectors often fail to transfer across different pre-trained models because their parameter spaces are misaligned. In this work, we show that successful transfer depends strongly on the gradient-sign structure of the new model. Based on this insight, we propose GradFix, which approximates the ideal sign structure and leverages it to transfer knowledge using only a handful of labeled samples. Notably, this requires no additional fine-tuning: we only compute a few target-model gradients without parameter updates and mask the source task vector accordingly. This yields an update that is locally aligned with the target loss landscape, effectively rebasing the task vector onto the new pre-training. We provide a theoretical guarantee that our method ensures first-order descent. Empirically, we demonstrate significant performance gains on vision and language benchmarks, consistently outperforming naive task vector addition and few-shot fine-tuning. We further show that transporting task vectors improves multi-task and multi-source model merging. Code is available at https://github.com/fillo-rinaldi/GradFix.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2510.09658")
get_code_for_paper("2510.09658")
have("2510.09658")

Connect an agent — have() is free.