ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model

• URL: http://arxiv.org/abs/2304.01116v1
• Date: Mon, 3 Apr 2023 16:29:00 GMT
• Title: ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model
• Authors: Mingyuan Zhang, Xinying Guo, Liang Pan, Zhongang Cai, Fangzhou Hong, Huirong Li, Lei Yang, Ziwei Liu
• Abstract summary: 3D human motion generation is crucial for creative industry. Recent advances rely on generative models with domain knowledge for text-driven motion generation. We propose ReMoDiffuse, a diffusion-model-based motion generation framework.
• Score: 33.64263969970544
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: 3D human motion generation is crucial for creative industry. Recent advances rely on generative models with domain knowledge for text-driven motion generation, leading to substantial progress in capturing common motions. However, the performance on more diverse motions remains unsatisfactory. In this work, we propose ReMoDiffuse, a diffusion-model-based motion generation framework that integrates a retrieval mechanism to refine the denoising process. ReMoDiffuse enhances the generalizability and diversity of text-driven motion generation with three key designs: 1) Hybrid Retrieval finds appropriate references from the database in terms of both semantic and kinematic similarities. 2) Semantic-Modulated Transformer selectively absorbs retrieval knowledge, adapting to the difference between retrieved samples and the target motion sequence. 3) Condition Mixture better utilizes the retrieval database during inference, overcoming the scale sensitivity in classifier-free guidance. Extensive experiments demonstrate that ReMoDiffuse outperforms state-of-the-art methods by balancing both text-motion consistency and motion quality, especially for more diverse motion generation.

Related papers

• MoRAG -- Multi-Fusion Retrieval Augmented Generation for Human Motion [8.94802080815133]
  MoRAG is a novel multi-part fusion based retrieval-augmented generation strategy for text-based human motion generation. We create diverse samples through the spatial composition of the retrieved motions. Our framework can serve as a plug-and-play module, improving the performance of motion diffusion models.
  arXiv  Detail & Related papers  (2024-09-18T17:03:30Z)
• Learning Generalizable Human Motion Generator with Reinforcement Learning [95.62084727984808]
  Text-driven human motion generation is one of the vital tasks in computer-aided content creation. Existing methods often overfit specific motion expressions in the training data, hindering their ability to generalize. We present textbfInstructMotion, which incorporate the trail and error paradigm in reinforcement learning for generalizable human motion generation.
  arXiv  Detail & Related papers  (2024-05-24T13:29:12Z)
• Motion-aware 3D Gaussian Splatting for Efficient Dynamic Scene Reconstruction [89.53963284958037]
  We propose a novel motion-aware enhancement framework for dynamic scene reconstruction. Specifically, we first establish a correspondence between 3D Gaussian movements and pixel-level flow. For the prevalent deformation-based paradigm that presents a harder optimization problem, a transient-aware deformation auxiliary module is proposed.
  arXiv  Detail & Related papers  (2024-03-18T03:46:26Z)
• MotionMix: Weakly-Supervised Diffusion for Controllable Motion Generation [19.999239668765885]
  MotionMix is a weakly-supervised diffusion model that leverages both noisy and unannotated motion sequences. Our framework consistently achieves state-of-the-art performances on text-to-motion, action-to-motion, and music-to-dance tasks.
  arXiv  Detail & Related papers  (2024-01-20T04:58:06Z)
• SemanticBoost: Elevating Motion Generation with Augmented Textual Cues [73.83255805408126]
  Our framework comprises a Semantic Enhancement module and a Context-Attuned Motion Denoiser (CAMD) The CAMD approach provides an all-encompassing solution for generating high-quality, semantically consistent motion sequences. Our experimental results demonstrate that SemanticBoost, as a diffusion-based method, outperforms auto-regressive-based techniques.
  arXiv  Detail & Related papers  (2023-10-31T09:58:11Z)
• DiverseMotion: Towards Diverse Human Motion Generation via Discrete Diffusion [70.33381660741861]
  We present DiverseMotion, a new approach for synthesizing high-quality human motions conditioned on textual descriptions. We show that our DiverseMotion achieves the state-of-the-art motion quality and competitive motion diversity.
  arXiv  Detail & Related papers  (2023-09-04T05:43:48Z)
• Priority-Centric Human Motion Generation in Discrete Latent Space [59.401128190423535]
  We introduce a Priority-Centric Motion Discrete Diffusion Model (M2DM) for text-to-motion generation. M2DM incorporates a global self-attention mechanism and a regularization term to counteract code collapse. We also present a motion discrete diffusion model that employs an innovative noise schedule, determined by the significance of each motion token.
  arXiv  Detail & Related papers  (2023-08-28T10:40:16Z)
• Controllable Motion Synthesis and Reconstruction with Autoregressive Diffusion Models [18.50942770933098]
  MoDiff is an autoregressive probabilistic diffusion model over motion sequences conditioned on control contexts of other modalities. Our model integrates a cross-modal Transformer encoder and a Transformer-based decoder, which are found effective in capturing temporal correlations in motion and control modalities.
  arXiv  Detail & Related papers  (2023-04-03T08:17:08Z)
• MoFusion: A Framework for Denoising-Diffusion-based Motion Synthesis [73.52948992990191]
  MoFusion is a new denoising-diffusion-based framework for high-quality conditional human motion synthesis. We present ways to introduce well-known kinematic losses for motion plausibility within the motion diffusion framework. We demonstrate the effectiveness of MoFusion compared to the state of the art on established benchmarks in the literature.
  arXiv  Detail & Related papers  (2022-12-08T18:59:48Z)
• MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model [35.32967411186489]
  MotionDiffuse is a diffusion model-based text-driven motion generation framework. It excels at modeling complicated data distribution and generating vivid motion sequences. It responds to fine-grained instructions on body parts, and arbitrary-length motion synthesis with time-varied text prompts.
  arXiv  Detail & Related papers  (2022-08-31T17:58:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.