4M: Massively Multimodal Masked Modeling

Roman Bachmann, Teresa Yeo, Amir Zamir, David Mizrahi, Oguzhan Fatih Kar, Mingfei Gao, Afshin Dehghan

arXiv · PDF · Open in the Atlas

Code that ran

We lifted 14 functions out of this paper's own repositories and ran 11 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.

Repositories linked to this paper

• apple/ml-4m — marked official in Papers with Code

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Current machine learning models for vision are often highly specialized and limited to a single modality and task. In contrast, recent large language models exhibit a wide range of capabilities, hinting at a possibility for similarly versatile models in computer vision. In this paper, we take a step in this direction and propose a multimodal training scheme called 4M. It consists of training a single unified Transformer encoder-decoder using a masked modeling objective across a wide range of input/output modalities -including text, images, geometric, and semantic modalities, as well as neural network feature maps. 4M achieves scalability by unifying the representation space of all modalities through mapping them into discrete tokens and performing multimodal masked modeling on a small randomized subset of tokens. 4M leads to models that exhibit several key capabilities: (1) they can perform a diverse set of vision tasks out of the box, (2) they excel when fine-tuned for unseen downstream tasks or new input modalities, and (3) they can function as a generative model that can be conditioned on arbitrary modalities, enabling a wide variety of expressive multimodal editing capabilities with remarkable flexibility. Through experimental analyses, we demonstrate the potential of 4M for training versatile and scalable foundation models for vision tasks, setting the stage for further exploration in multimodal learning for vision and other domains. † For clarity, "modalities" usually denote the inputs to a model (e.g. sensory signals), and "tasks" usually denote the outputs (e.g. semantics). Our method enables a symmetric input-output structure, thus we use "modalities" and "tasks" interchangeably in this paper. * Equal contribution & corresponding authors. 37th Conference on Neural Information Processing Systems (NeurIPS 2023).

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2312.06647")
get_code_for_paper("2312.06647")
have("2312.06647")

Connect an agent — have() is free.