FG-MDM: Towards Zero-Shot Human Motion Generation via ChatGPT-Refined Descriptions

• URL: http://arxiv.org/abs/2312.02772v3
• Date: Thu, 05 Dec 2024 14:19:26 GMT
• Title: FG-MDM: Towards Zero-Shot Human Motion Generation via ChatGPT-Refined Descriptions
• Authors: Xu Shi, Wei Yao, Chuanchen Luo, Junran Peng, Hongwen Zhang, Yunlian Sun,
• Abstract summary: We propose a new framework named Fine-Grained Human Motion Diffusion Model (FG-MDM) for zero-shot human motion generation.<n> Specifically, we first parse previous vague textual annotations into fine-grained descriptions of different body parts.<n>FG-MDM can generate human motions beyond the scope of original datasets owing to descriptions that are closer to motion essence.
• Score: 19.695991127631974
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recently, significant progress has been made in text-based motion generation, enabling the generation of diverse and high-quality human motions that conform to textual descriptions. However, generating motions beyond the distribution of original datasets remains challenging, i.e., zero-shot generation. By adopting a divide-and-conquer strategy, we propose a new framework named Fine-Grained Human Motion Diffusion Model (FG-MDM) for zero-shot human motion generation. Specifically, we first parse previous vague textual annotations into fine-grained descriptions of different body parts by leveraging a large language model. We then use these fine-grained descriptions to guide a transformer-based diffusion model, which further adopts a design of part tokens. FG-MDM can generate human motions beyond the scope of original datasets owing to descriptions that are closer to motion essence. Our experimental results demonstrate the superiority of FG-MDM over previous methods in zero-shot settings. We will release our fine-grained textual annotations for HumanML3D and KIT.

Related papers

• GENMO: A GENeralist Model for Human MOtion [64.16188966024542]
  We present GENMO, a unified Generalist Model for Human Motion that bridges motion estimation and generation in a single framework.<n>Our key insight is to reformulate motion estimation as constrained motion generation, where the output motion must precisely satisfy observed conditioning signals.<n>Our novel architecture handles variable-length motions and mixed multimodal conditions (text, audio, video) at different time intervals, offering flexible control.
  arXiv  Detail & Related papers  (2025-05-02T17:59:55Z)
• PackDiT: Joint Human Motion and Text Generation via Mutual Prompting [22.53146582495341]
  PackDiT is the first diffusion-based generative model capable of performing various tasks simultaneously. We train PackDiT on the HumanML3D dataset, achieving state-of-the-art text-to-motion performance with an FID score of 0.106. Our experiments further demonstrate that diffusion models are effective for motion-to-text generation, achieving performance comparable to that of autoregressive models.
  arXiv  Detail & Related papers  (2025-01-27T22:51:45Z)
• Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordance [48.986552871497]
  We introduce a novel two-stage framework that employs scene affordance as an intermediate representation. By leveraging scene affordance maps, our method overcomes the difficulty in generating human motion under multimodal condition signals. Our approach consistently outperforms all baselines on established benchmarks, including HumanML3D and HUMANISE.
  arXiv  Detail & Related papers  (2024-03-26T18:41:07Z)
• Motion Generation from Fine-grained Textual Descriptions [29.033358642532722]
  We build a large-scale language-motion dataset specializing in fine-grained textual descriptions, FineHumanML3D. We design a new text2motion model, FineMotionDiffuse, making full use of fine-grained textual information. Our evaluation shows that FineMotionDiffuse trained on FineHumanML3D improves FID by a large margin of 0.38, compared with competitive baselines.
  arXiv  Detail & Related papers  (2024-03-20T11:38:30Z)
• OMG: Towards Open-vocabulary Motion Generation via Mixture of Controllers [45.808597624491156]
  We present OMG, a novel framework, which enables compelling motion generation from zero-shot open-vocabulary text prompts. At the pre-training stage, our model improves the generation ability by learning the rich out-of-domain inherent motion traits. At the fine-tuning stage, we introduce motion ControlNet, which incorporates text prompts as conditioning information.
  arXiv  Detail & Related papers  (2023-12-14T14:31:40Z)
• Hierarchical Generation of Human-Object Interactions with Diffusion Probabilistic Models [71.64318025625833]
  This paper presents a novel approach to generating the 3D motion of a human interacting with a target object. Our framework first generates a set of milestones and then synthesizes the motion along them. The experiments on the NSM, COUCH, and SAMP datasets show that our approach outperforms previous methods by a large margin in both quality and diversity.
  arXiv  Detail & Related papers  (2023-10-03T17:50:23Z)
• DiverseMotion: Towards Diverse Human Motion Generation via Discrete Diffusion [70.33381660741861]
  We present DiverseMotion, a new approach for synthesizing high-quality human motions conditioned on textual descriptions. We show that our DiverseMotion achieves the state-of-the-art motion quality and competitive motion diversity.
  arXiv  Detail & Related papers  (2023-09-04T05:43:48Z)
• Priority-Centric Human Motion Generation in Discrete Latent Space [59.401128190423535]
  We introduce a Priority-Centric Motion Discrete Diffusion Model (M2DM) for text-to-motion generation. M2DM incorporates a global self-attention mechanism and a regularization term to counteract code collapse. We also present a motion discrete diffusion model that employs an innovative noise schedule, determined by the significance of each motion token.
  arXiv  Detail & Related papers  (2023-08-28T10:40:16Z)
• Human Motion Diffusion Model [35.05219668478535]
  Motion Diffusion Model (MDM) is a transformer-based generative model for the human motion domain. We show that our model is trained with lightweight resources and yet achieves state-of-the-art results on leading benchmarks for text-to-motion and action-to-motion.
  arXiv  Detail & Related papers  (2022-09-29T16:27:53Z)
• REMOT: A Region-to-Whole Framework for Realistic Human Motion Transfer [96.64111294772141]
  Human Video Motion Transfer (HVMT) aims to, given an image of a source person, generate his/her video that imitates the motion of the driving person. Existing methods for HVMT mainly exploit Generative Adversarial Networks (GANs) to perform the warping operation. This paper presents a novel REgionto-whole human MOtion Transfer framework based on GANs.
  arXiv  Detail & Related papers  (2022-09-01T14:03:51Z)
• MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model [35.32967411186489]
  MotionDiffuse is a diffusion model-based text-driven motion generation framework. It excels at modeling complicated data distribution and generating vivid motion sequences. It responds to fine-grained instructions on body parts, and arbitrary-length motion synthesis with time-varied text prompts.
  arXiv  Detail & Related papers  (2022-08-31T17:58:54Z)
• TEMOS: Generating diverse human motions from textual descriptions [53.85978336198444]
  We address the problem of generating diverse 3D human motions from textual descriptions. We propose TEMOS, a text-conditioned generative model leveraging variational autoencoder (VAE) training with human motion data. We show that TEMOS framework can produce both skeleton-based animations as in prior work, as well more expressive SMPL body motions.
  arXiv  Detail & Related papers  (2022-04-25T14:53:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.