Motion Mamba: Efficient and Long Sequence Motion Generation

• URL: http://arxiv.org/abs/2403.07487v4
• Date: Sat, 3 Aug 2024 07:48:15 GMT
• Title: Motion Mamba: Efficient and Long Sequence Motion Generation
• Authors: Zeyu Zhang, Akide Liu, Ian Reid, Richard Hartley, Bohan Zhuang, Hao Tang,
• Abstract summary: Recent advancements in state space models (SSMs) have showcased considerable promise in long sequence modeling. We propose Motion Mamba, a simple and efficient approach that presents the pioneering motion generation model utilized SSMs. Our proposed method achieves up to 50% FID improvement and up to 4 times faster on the HumanML3D and KIT-ML datasets.
• Score: 26.777455596989526
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Human motion generation stands as a significant pursuit in generative computer vision, while achieving long-sequence and efficient motion generation remains challenging. Recent advancements in state space models (SSMs), notably Mamba, have showcased considerable promise in long sequence modeling with an efficient hardware-aware design, which appears to be a promising direction to build motion generation model upon it. Nevertheless, adapting SSMs to motion generation faces hurdles since the lack of a specialized design architecture to model motion sequence. To address these challenges, we propose Motion Mamba, a simple and efficient approach that presents the pioneering motion generation model utilized SSMs. Specifically, we design a Hierarchical Temporal Mamba (HTM) block to process temporal data by ensemble varying numbers of isolated SSM modules across a symmetric U-Net architecture aimed at preserving motion consistency between frames. We also design a Bidirectional Spatial Mamba (BSM) block to bidirectionally process latent poses, to enhance accurate motion generation within a temporal frame. Our proposed method achieves up to 50% FID improvement and up to 4 times faster on the HumanML3D and KIT-ML datasets compared to the previous best diffusion-based method, which demonstrates strong capabilities of high-quality long sequence motion modeling and real-time human motion generation. See project website https://steve-zeyu-zhang.github.io/MotionMamba/

Related papers

• Quo Vadis, Motion Generation? From Large Language Models to Large Motion Models [70.78051873517285]
  We present MotionBase, the first million-level motion generation benchmark. By leveraging this vast dataset, our large motion model demonstrates strong performance across a broad range of motions. We introduce a novel 2D lookup-free approach for motion tokenization, which preserves motion information and expands codebook capacity.
  arXiv  Detail & Related papers  (2024-10-04T10:48:54Z)
• Temporal and Interactive Modeling for Efficient Human-Human Motion Generation [30.857021853999644]
  We introduce TIM (Temporal and Interactive Modeling), an efficient and effective approach that presents the pioneering human-human motion generation model. Specifically, we first propose Causal Interactive Injection to leverage the temporal properties of motion sequences and avoid non-causal and cumbersome modeling. Finally, to generate smoother and more rational motion, we design Localized Pattern Amplification to capture short-term motion patterns.
  arXiv  Detail & Related papers  (2024-08-30T09:22:07Z)
• InfiniMotion: Mamba Boosts Memory in Transformer for Arbitrary Long Motion Generation [31.775481455602634]
  Current methods struggle to handle long motion sequences as a single input due to high computational cost. We propose InfiniMotion, a method that generates continuous motion sequences of arbitrary length within an autoregressive framework. We highlight its groundbreaking capability by generating a continuous 1-hour human motion with around 80,000 frames.
  arXiv  Detail & Related papers  (2024-07-14T03:12:19Z)
• SMCD: High Realism Motion Style Transfer via Mamba-based Diffusion [12.426879081036116]
  Style transfer is widely applied in multimedia scenarios such as movies, games, and the Metaverse. Most of the current work in this field adopts the GAN, which may lead to instability and convergence issues. We propose the Style Motion Conditioned Diffusion (SMCD) framework for the first time, which can more comprehensively learn the style features of motion.
  arXiv  Detail & Related papers  (2024-05-05T08:28:07Z)
• Large Motion Model for Unified Multi-Modal Motion Generation [50.56268006354396]
  Large Motion Model (LMM) is a motion-centric, multi-modal framework that unifies mainstream motion generation tasks into a generalist model. LMM tackles these challenges from three principled aspects.
  arXiv  Detail & Related papers  (2024-04-01T17:55:11Z)
• MambaMixer: Efficient Selective State Space Models with Dual Token and Channel Selection [5.37935922811333]
  MambaMixer is a new architecture with data-dependent weights that uses a dual selection mechanism across tokens and channels. As a proof of concept, we design Vision MambaMixer (ViM2) and Time Series MambaMixer (TSM2) architectures based on the MambaMixer block.
  arXiv  Detail & Related papers  (2024-03-29T00:05:13Z)
• Spectral Motion Alignment for Video Motion Transfer using Diffusion Models [54.32923808964701]
  Spectral Motion Alignment (SMA) is a framework that refines and aligns motion vectors using Fourier and wavelet transforms. SMA learns motion patterns by incorporating frequency-domain regularization, facilitating the learning of whole-frame global motion dynamics. Extensive experiments demonstrate SMA's efficacy in improving motion transfer while maintaining computational efficiency and compatibility across various video customization frameworks.
  arXiv  Detail & Related papers  (2024-03-22T14:47:18Z)
• FineMoGen: Fine-Grained Spatio-Temporal Motion Generation and Editing [56.29102849106382]
  FineMoGen is a diffusion-based motion generation and editing framework. It can synthesize fine-grained motions, with spatial-temporal composition to the user instructions. FineMoGen further enables zero-shot motion editing capabilities with the aid of modern large language models.
  arXiv  Detail & Related papers  (2023-12-22T16:56:02Z)
• Motion Flow Matching for Human Motion Synthesis and Editing [75.13665467944314]
  We propose emphMotion Flow Matching, a novel generative model for human motion generation featuring efficient sampling and effectiveness in motion editing applications. Our method reduces the sampling complexity from thousand steps in previous diffusion models to just ten steps, while achieving comparable performance in text-to-motion and action-to-motion generation benchmarks.
  arXiv  Detail & Related papers  (2023-12-14T12:57:35Z)
• MotionTrack: Learning Motion Predictor for Multiple Object Tracking [68.68339102749358]
  We introduce a novel motion-based tracker, MotionTrack, centered around a learnable motion predictor. Our experimental results demonstrate that MotionTrack yields state-of-the-art performance on datasets such as Dancetrack and SportsMOT.
  arXiv  Detail & Related papers  (2023-06-05T04:24:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.