Rethinking Diffusion for Text-Driven Human Motion Generation

• URL: http://arxiv.org/abs/2411.16575v1
• Date: Mon, 25 Nov 2024 16:59:42 GMT
• Title: Rethinking Diffusion for Text-Driven Human Motion Generation
• Authors: Zichong Meng, Yiming Xie, Xiaogang Peng, Zeyu Han, Huaizu Jiang,
• Abstract summary: Since 2023, Vector Quantization (VQ)-based discrete generation methods have dominated human motion generation. In this work, we investigate why current VQ-based methods perform well and explore the limitations of existing diffusion-based methods. Our approach introduces a human motion diffusion model enabled to perform bidirectional masked autoregression.
• Score: 8.153961351540834
• License:
• Abstract: Since 2023, Vector Quantization (VQ)-based discrete generation methods have rapidly dominated human motion generation, primarily surpassing diffusion-based continuous generation methods in standard performance metrics. However, VQ-based methods have inherent limitations. Representing continuous motion data as limited discrete tokens leads to inevitable information loss, reduces the diversity of generated motions, and restricts their ability to function effectively as motion priors or generation guidance. In contrast, the continuous space generation nature of diffusion-based methods makes them well-suited to address these limitations and with even potential for model scalability. In this work, we systematically investigate why current VQ-based methods perform well and explore the limitations of existing diffusion-based methods from the perspective of motion data representation and distribution. Drawing on these insights, we preserve the inherent strengths of a diffusion-based human motion generation model and gradually optimize it with inspiration from VQ-based approaches. Our approach introduces a human motion diffusion model enabled to perform bidirectional masked autoregression, optimized with a reformed data representation and distribution. Additionally, we also propose more robust evaluation methods to fairly assess different-based methods. Extensive experiments on benchmark human motion generation datasets demonstrate that our method excels previous methods and achieves state-of-the-art performances.

Related papers

• G2D2: Gradient-guided Discrete Diffusion for image inverse problem solving [55.185588994883226]
  This paper presents a novel method for addressing linear inverse problems by leveraging image-generation models based on discrete diffusion as priors. To the best of our knowledge, this is the first approach to use discrete diffusion model-based priors for solving image inverse problems.
  arXiv  Detail & Related papers  (2024-10-09T06:18:25Z)
• Text-to-Image Rectified Flow as Plug-and-Play Priors [52.586838532560755]
  Rectified flow is a novel class of generative models that enforces a linear progression from the source to the target distribution. We show that rectified flow approaches surpass in terms of generation quality and efficiency, requiring fewer inference steps. Our method also displays competitive performance in image inversion and editing.
  arXiv  Detail & Related papers  (2024-06-05T14:02:31Z)
• Neural Flow Diffusion Models: Learnable Forward Process for Improved Diffusion Modelling [2.1779479916071067]
  We introduce a novel framework that enhances diffusion models by supporting a broader range of forward processes. We also propose a novel parameterization technique for learning the forward process. Results underscore NFDM's versatility and its potential for a wide range of applications.
  arXiv  Detail & Related papers  (2024-04-19T15:10:54Z)
• Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation [59.184980778643464]
  Fine-tuning Diffusion Models remains an underexplored frontier in generative artificial intelligence (GenAI) In this paper, we introduce an innovative technique called self-play fine-tuning for diffusion models (SPIN-Diffusion) Our approach offers an alternative to conventional supervised fine-tuning and RL strategies, significantly improving both model performance and alignment.
  arXiv  Detail & Related papers  (2024-02-15T18:59:18Z)
• HuTuMotion: Human-Tuned Navigation of Latent Motion Diffusion Models with Minimal Feedback [46.744192144648764]
  HuTuMotion is an innovative approach for generating natural human motions that navigates latent motion diffusion models by leveraging few-shot human feedback. Our findings reveal that utilizing few-shot feedback can yield performance levels on par with those attained through extensive human feedback.
  arXiv  Detail & Related papers  (2023-12-19T15:13:08Z)
• Fast Sampling via Discrete Non-Markov Diffusion Models [49.598085130313514]
  We propose a discrete non-Markov diffusion model, which admits an accelerated reverse sampling for discrete data generation. Our method significantly reduces the number of function evaluations (i.e., calls to the neural network), making the sampling process much faster.
  arXiv  Detail & Related papers  (2023-12-14T18:14:11Z)
• Unsupervised Discovery of Interpretable Directions in h-space of Pre-trained Diffusion Models [63.1637853118899]
  We propose the first unsupervised and learning-based method to identify interpretable directions in h-space of pre-trained diffusion models. We employ a shift control module that works on h-space of pre-trained diffusion models to manipulate a sample into a shifted version of itself. By jointly optimizing them, the model will spontaneously discover disentangled and interpretable directions.
  arXiv  Detail & Related papers  (2023-10-15T18:44:30Z)
• STGlow: A Flow-based Generative Framework with Dual Graphormer for Pedestrian Trajectory Prediction [22.553356096143734]
  We propose a novel generative flow based framework with dual graphormer for pedestrian trajectory prediction (STGlow) Our method can more precisely model the underlying data distribution by optimizing the exact log-likelihood of motion behaviors. Experimental results on several benchmarks demonstrate that our method achieves much better performance compared to previous state-of-the-art approaches.
  arXiv  Detail & Related papers  (2022-11-21T07:29:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.