FreeBind: Free Lunch in Unified Multimodal Space via Knowledge Fusion

Yang Zhao, Zhou Zhao, Peng Gao, Luping Liu, Rongjie Huang, Zehan Wang, Haifeng Huang, Ziang Zhang, Zhenhui Ye, Tao Jin, Xize Cheng

arXiv · PDF · Open in the Atlas

Code that ran

We lifted 2 functions out of this paper's own repositories and ran 2 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.

Repositories linked to this paper

• zehanwang01/freebind — marked official in Papers with Code; named in the paper
• zehanwang01/FreeBind

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Unified multi-model representation spaces are the foundation of multimodal understanding and generation. However, the billions of model parameters and catastrophic forgetting problems make it challenging to further enhance pre-trained unified spaces. In this work, we propose FreeBind, an idea that treats multimodal representation spaces as basic units, and freely augments pre-trained unified space by integrating knowledge from extra expert spaces via "space bonds". Specifically, we introduce two kinds of basic space bonds: 1) Space Displacement Bond and 2) Space Combination Bond. Based on these basic bonds, we design Complex Sequential & Parallel Bonds to effectively integrate multiple spaces simultaneously. Benefiting from the modularization concept, we further propose a coarse-to-fine customized inference strategy to flexibly adjust the enhanced unified space for different purposes. Experimentally, we bind ImageBind with extra image-text and audio-text expert spaces, resulting in three main variants: ImageBind++, InternVL IB and InternVL IB ++. These resulting spaces outperform ImageBind on 5 audio-image-text downstream tasks across 9 datasets. Moreover, via customized inference, it even surpasses the advanced audio-text and image-text expert spaces. Our code and checkpoints will be released at https:// github.com/zehanwang01/FreeBind

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2405.04883")
get_code_for_paper("2405.04883")
have("2405.04883")

Connect an agent — have() is free.