M 3 GPT: An Advanced Multimodal, Multitask Framework for Motion Comprehension and Generation

Yaowei Wang, Shiguang Shan, Zhuo Li, Mingshuang Luo, Ruibing Hou, Hong Chang, Zimo Liu

arXiv · PDF · Open in the Atlas

Code that ran

We lifted 25 functions out of this paper's own repositories and ran 17 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.

Repositories linked to this paper

• luomingshuang/m3gpt — marked official in Papers with Code; named in the paper
• luomingshuang/M3GPT

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

This paper presents M 3 GPT, an advanced Multimodal, Multitask framework for Motion comprehension and generation. M 3 GPT operates on three fundamental principles. The first focuses on creating a unified representation space for various motion-relevant modalities. We employ discrete vector quantization for multimodal conditional signals, such as text, music and motion/dance, enabling seamless integration into a large language model (LLM) with a single vocabulary. The second involves modeling motion generation directly in the raw motion space. This strategy circumvents the information loss associated with a discrete tokenizer, resulting in more detailed and comprehensive motion generation. Third, M 3 GPT learns to model the connections and synergies among various motion-relevant tasks. Text, the most familiar and well-understood modality for LLMs, is utilized as a bridge to establish connections between different motion tasks, facilitating mutual reinforcement. To our knowledge, M 3 GPT is the first model capable of comprehending and generating motions based on multiple signals. Extensive experiments highlight M 3 GPT's superior performance across various motion-relevant tasks and its powerful zero-shot generalization capabilities for extremely challenging tasks. Project page: https://github.com/luomingshuang/M3GPT. Can you generate a motion for the caption that a person performs a knee tuck to kick L? Can you generate a music for the dance? Can you fullfill the following motion? A person is performing a Hip Bounce Wrist Circle movement. Can you generate a dance for the music? Can you translate the motion into a caption? M 3 GPT Can you predict a motion for a given motion?

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2405.16273")
get_code_for_paper("2405.16273")
have("2405.16273")

Connect an agent — have() is free.