MotionGPT-2: A General-Purpose Motion-Language Model for Motion Generation and Understanding

• URL: http://arxiv.org/abs/2410.21747v1
• Date: Tue, 29 Oct 2024 05:25:34 GMT
• Title: MotionGPT-2: A General-Purpose Motion-Language Model for Motion Generation and Understanding
• Authors: Yuan Wang, Di Huang, Yaqi Zhang, Wanli Ouyang, Jile Jiao, Xuetao Feng, Yan Zhou, Pengfei Wan, Shixiang Tang, Dan Xu,
• Abstract summary: MotionGPT-2 is a unified Large Motion-Language Model (LMLMLM) It supports multimodal control conditions through pre-trained Large Language Models (LLMs) It is highly adaptable to the challenging 3D holistic motion generation task.
• Score: 76.30210465222218
• License:
• Abstract: Generating lifelike human motions from descriptive texts has experienced remarkable research focus in the recent years, propelled by the emerging requirements of digital humans.Despite impressive advances, existing approaches are often constrained by limited control modalities, task specificity, and focus solely on body motion representations.In this paper, we present MotionGPT-2, a unified Large Motion-Language Model (LMLM) that addresses these limitations. MotionGPT-2 accommodates multiple motion-relevant tasks and supporting multimodal control conditions through pre-trained Large Language Models (LLMs). It quantizes multimodal inputs-such as text and single-frame poses-into discrete, LLM-interpretable tokens, seamlessly integrating them into the LLM's vocabulary. These tokens are then organized into unified prompts, guiding the LLM to generate motion outputs through a pretraining-then-finetuning paradigm. We also show that the proposed MotionGPT-2 is highly adaptable to the challenging 3D holistic motion generation task, enabled by the innovative motion discretization framework, Part-Aware VQVAE, which ensures fine-grained representations of body and hand movements. Extensive experiments and visualizations validate the effectiveness of our method, demonstrating the adaptability of MotionGPT-2 across motion generation, motion captioning, and generalized motion completion tasks.

Related papers

• MotionCraft: Crafting Whole-Body Motion with Plug-and-Play Multimodal Controls [30.487510829107908]
  We propose MotionCraft, a unified diffusion transformer that crafts whole-body motion with plug-and-play multimodal control. Our framework employs a coarse-to-fine training strategy, starting with the first stage of text-to-motion semantic pre-training. We introduce MC-Bench, the first available multimodal whole-body motion generation benchmark based on the unified SMPL-X format.
  arXiv  Detail & Related papers  (2024-07-30T18:57:06Z)
• Motion-Agent: A Conversational Framework for Human Motion Generation with LLMs [67.59291068131438]
  Motion-Agent is a conversational framework designed for general human motion generation, editing, and understanding. Motion-Agent employs an open-source pre-trained language model to develop a generative agent, MotionLLM, that bridges the gap between motion and text.
  arXiv  Detail & Related papers  (2024-05-27T09:57:51Z)
• FreeMotion: A Unified Framework for Number-free Text-to-Motion Synthesis [65.85686550683806]
  This paper reconsiders motion generation and proposes to unify the single and multi-person motion by the conditional motion distribution. Based on our framework, the current single-person motion spatial control method could be seamlessly integrated, achieving precise control of multi-person motion.
  arXiv  Detail & Related papers  (2024-05-24T17:57:57Z)
• Animate Your Motion: Turning Still Images into Dynamic Videos [58.63109848837741]
  We introduce Scene and Motion Conditional Diffusion (SMCD), a novel methodology for managing multimodal inputs. SMCD incorporates a recognized motion conditioning module and investigates various approaches to integrate scene conditions. Our design significantly enhances video quality, motion precision, and semantic coherence.
  arXiv  Detail & Related papers  (2024-03-15T10:36:24Z)
• MoConVQ: Unified Physics-Based Motion Control via Scalable Discrete Representations [25.630268570049708]
  MoConVQ is a novel unified framework for physics-based motion control leveraging scalable discrete representations. Our approach effectively learns motion embeddings from a large, unstructured dataset spanning tens of hours of motion examples.
  arXiv  Detail & Related papers  (2023-10-16T09:09:02Z)
• DiverseMotion: Towards Diverse Human Motion Generation via Discrete Diffusion [70.33381660741861]
  We present DiverseMotion, a new approach for synthesizing high-quality human motions conditioned on textual descriptions. We show that our DiverseMotion achieves the state-of-the-art motion quality and competitive motion diversity.
  arXiv  Detail & Related papers  (2023-09-04T05:43:48Z)
• Priority-Centric Human Motion Generation in Discrete Latent Space [59.401128190423535]
  We introduce a Priority-Centric Motion Discrete Diffusion Model (M2DM) for text-to-motion generation. M2DM incorporates a global self-attention mechanism and a regularization term to counteract code collapse. We also present a motion discrete diffusion model that employs an innovative noise schedule, determined by the significance of each motion token.
  arXiv  Detail & Related papers  (2023-08-28T10:40:16Z)
• MotionGPT: Human Motion as a Foreign Language [47.21648303282788]
  Human motion displays a semantic coupling akin to human language, often perceived as a form of body language. By fusing language data with large-scale motion models, motion-language pre-training can enhance the performance of motion-related tasks. We propose MotionGPT, a unified, versatile, and user-friendly motion-language model to handle multiple motion-relevant tasks.
  arXiv  Detail & Related papers  (2023-06-26T15:53:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.