We lifted 1 functions out of this paper's own repositories and ran 0 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| copy not recorded | — | 0 of 1 |
| Function | Status | Where it lives |
|---|---|---|
| fact_forward_attn | Not yet run | this paper's copy was not recorded; identical code first harvested from jieshibo/petl-vit pointer only · get_code("6ba8f0e9af871ebc") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Recent work has explored the potential to adapt a pre-trained vision transformer (ViT) by updating only a few parameters so as to improve storage efficiency, called parameter-efficient transfer learning (PETL). Current PETL methods have shown that by tuning only 0.5% of the parameters, ViT can be adapted to downstream tasks with even better performance than full fine-tuning. In this paper, we aim to further promote the efficiency of PETL to meet the extreme storage constraint in real-world applications. To this end, we propose a tensorization-decomposition framework to store the weight increments, in which the weights of each ViT are tensorized into a single 3D tensor, and their increments are then decomposed into lightweight factors. In the fine-tuning process, only the factors need to be updated and stored, termed Factor-Tuning (FacT). On VTAB-1K benchmark, our method performs on par with NOAH, the state-of-the-art PETL method, while being 5x more parameter-efficient. We also present a tiny version that only uses 8K (0.01% of ViT's parameters) trainable parameters but outperforms full fine-tuning and many other PETL methods such as VPT and BitFit. In few-shot settings, FacT also beats all PETL baselines using the fewest parameters, demonstrating its strong capability in the low-data regime.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2212.03145")
get_code_for_paper("2212.03145")
have("2212.03145")
Connect an agent — have() is free.