MotionBank: A Large-scale Video Motion Benchmark with Disentangled Rule-based Annotations

• URL: http://arxiv.org/abs/2410.13790v1
• Date: Thu, 17 Oct 2024 17:31:24 GMT
• Title: MotionBank: A Large-scale Video Motion Benchmark with Disentangled Rule-based Annotations
• Authors: Liang Xu, Shaoyang Hua, Zili Lin, Yifan Liu, Feipeng Ma, Yichao Yan, Xin Jin, Xiaokang Yang, Wenjun Zeng,
• Abstract summary: We build MotionBank, which comprises 13 video action datasets, 1.24M motion sequences, and 132.9M frames of natural and diverse human motions. Our MotionBank is beneficial for general motion-related tasks of human motion generation, motion in-context generation, and motion understanding.
• Score: 85.85596165472663
• License:
• Abstract: In this paper, we tackle the problem of how to build and benchmark a large motion model (LMM). The ultimate goal of LMM is to serve as a foundation model for versatile motion-related tasks, e.g., human motion generation, with interpretability and generalizability. Though advanced, recent LMM-related works are still limited by small-scale motion data and costly text descriptions. Besides, previous motion benchmarks primarily focus on pure body movements, neglecting the ubiquitous motions in context, i.e., humans interacting with humans, objects, and scenes. To address these limitations, we consolidate large-scale video action datasets as knowledge banks to build MotionBank, which comprises 13 video action datasets, 1.24M motion sequences, and 132.9M frames of natural and diverse human motions. Different from laboratory-captured motions, in-the-wild human-centric videos contain abundant motions in context. To facilitate better motion text alignment, we also meticulously devise a motion caption generation algorithm to automatically produce rule-based, unbiased, and disentangled text descriptions via the kinematic characteristics for each motion. Extensive experiments show that our MotionBank is beneficial for general motion-related tasks of human motion generation, motion in-context generation, and motion understanding. Video motions together with the rule-based text annotations could serve as an efficient alternative for larger LMMs. Our dataset, codes, and benchmark will be publicly available at https://github.com/liangxuy/MotionBank.

Related papers

• MotionAgent: Fine-grained Controllable Video Generation via Motion Field Agent [58.09607975296408]
  We propose MotionAgent, enabling fine-grained motion control for text-guided image-to-video generation. The key technique is the motion field agent that converts motion information in text prompts into explicit motion fields. We construct a subset of VBench to evaluate the alignment of motion information in the text and the generated video, outperforming other advanced models on motion generation accuracy.
  arXiv  Detail & Related papers  (2025-02-05T14:26:07Z)
• MotionLab: Unified Human Motion Generation and Editing via the Motion-Condition-Motion Paradigm [6.920041357348772]
  Human motion generation and editing are key components of computer graphics and vision. We introduce a novel paradigm: Motion-Condition-Motion, which enables the unified formulation of diverse tasks. Based on this paradigm, we propose a unified framework, MotionLab, which incorporates rectified flows to learn the mapping from source motion to target motion.
  arXiv  Detail & Related papers  (2025-02-04T14:43:26Z)
• Motion Prompting: Controlling Video Generation with Motion Trajectories [57.049252242807874]
  We train a video generation model conditioned on sparse or dense video trajectories. We translate high-level user requests into detailed, semi-dense motion prompts. We demonstrate our approach through various applications, including camera and object motion control, "interacting" with an image, motion transfer, and image editing.
  arXiv  Detail & Related papers  (2024-12-03T18:59:56Z)
• MotionGPT-2: A General-Purpose Motion-Language Model for Motion Generation and Understanding [76.30210465222218]
  MotionGPT-2 is a unified Large Motion-Language Model (LMLMLM) It supports multimodal control conditions through pre-trained Large Language Models (LLMs) It is highly adaptable to the challenging 3D holistic motion generation task.
  arXiv  Detail & Related papers  (2024-10-29T05:25:34Z)
• DART: A Diffusion-Based Autoregressive Motion Model for Real-Time Text-Driven Motion Control [12.465927271402442]
  Text-conditioned human motion generation allows for user interaction through natural language. DART is a Diffusion-based Autoregressive motion primitive model for Real-time Text-driven motion control. We present effective algorithms for both approaches, demonstrating our model's versatility and superior performance in various motion synthesis tasks.
  arXiv  Detail & Related papers  (2024-10-07T17:58:22Z)
• MotionLLM: Understanding Human Behaviors from Human Motions and Videos [40.132643319573205]
  This study delves into the realm of multi-modality (i.e., video and motion modalities) human behavior understanding. We present MotionLLM, a framework for human motion understanding, captioning, and reasoning.
  arXiv  Detail & Related papers  (2024-05-30T17:59:50Z)
• MotionCrafter: One-Shot Motion Customization of Diffusion Models [66.44642854791807]
  We introduce MotionCrafter, a one-shot instance-guided motion customization method. MotionCrafter employs a parallel spatial-temporal architecture that injects the reference motion into the temporal component of the base model. During training, a frozen base model provides appearance normalization, effectively separating appearance from motion.
  arXiv  Detail & Related papers  (2023-12-08T16:31:04Z)
• DiverseMotion: Towards Diverse Human Motion Generation via Discrete Diffusion [70.33381660741861]
  We present DiverseMotion, a new approach for synthesizing high-quality human motions conditioned on textual descriptions. We show that our DiverseMotion achieves the state-of-the-art motion quality and competitive motion diversity.
  arXiv  Detail & Related papers  (2023-09-04T05:43:48Z)
• MotionGPT: Human Motion as a Foreign Language [47.21648303282788]
  Human motion displays a semantic coupling akin to human language, often perceived as a form of body language. By fusing language data with large-scale motion models, motion-language pre-training can enhance the performance of motion-related tasks. We propose MotionGPT, a unified, versatile, and user-friendly motion-language model to handle multiple motion-relevant tasks.
  arXiv  Detail & Related papers  (2023-06-26T15:53:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.