M 2 PT: Multimodal Prompt Tuning for Zero-shot Instruction Learning

Zenglin Xu, Shaoliang Nie, James Liang, Dongfang Liu, Qifan Wang, Jiahao Liu, Yuning Mao, Yiming Cui, Fuli Feng, Cheng Han, Lifu Huang, Yiyang Liu, and 2 more

arXiv · PDF · Open in the Atlas

Code that ran

We lifted 26 functions out of this paper's own repositories and ran 16 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.

Repositories linked to this paper

• william-wang618/m2pt — marked official in Papers with Code; named in the paper
• william-wang618/mmpt-emnlp2024 — marked official in Papers with Code; named in the paper
• BradyFU/Awesome-Multimodal-Large-Language-Models

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Multimodal Large Language Models (MLLMs) demonstrate remarkable performance across a wide range of domains, with increasing emphasis on enhancing their zero-shot generalization capabilities for unseen tasks across various modalities. Instruction tuning has emerged as an effective strategy for achieving zeroshot generalization by finetuning pretrained models on diverse multimodal tasks. As the scale of MLLMs continues to grow, parameterefficient finetuning becomes increasingly critical. However, most existing parameter-efficient approaches focus only on single modalities and often overlook the multimodal characteristics during finetuning. In this work, we introduce a novel Multimodal Prompt Tuning (M 2 PT) approach for efficient instruction tuning of MLLMs. M 2 PT effectively integrates visual and textual prompts into the vision encoder and language processor respectively during finetuning, facilitating the extraction and alignment of features across modalities. Empirical results on various multimodal evaluation datasets demonstrate the superior performance of our approach compared to several state-of-the-art baselines. A comprehensive set of ablation studies validates the effectiveness of our prompt design and the efficiency of our approach.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2409.15657")
get_code_for_paper("2409.15657")
have("2409.15657")

Connect an agent — have() is free.