MotionScript: Natural Language Descriptions for Expressive 3D Human Motions

• URL: http://arxiv.org/abs/2312.12634v4
• Date: Sun, 16 Mar 2025 17:50:27 GMT
• Title: MotionScript: Natural Language Descriptions for Expressive 3D Human Motions
• Authors: Payam Jome Yazdian, Rachel Lagasse, Hamid Mohammadi, Eric Liu, Li Cheng, Angelica Lim,
• Abstract summary: We introduce MotionScript, a novel framework for generating highly detailed, natural language descriptions of 3D human motions.<n>MotionScript provides fine-grained, structured descriptions that capture the full complexity of human movement.<n> MotionScript serves as both a descriptive tool and a training resource for text-to-motion models.
• Score: 8.050271017133076
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce MotionScript, a novel framework for generating highly detailed, natural language descriptions of 3D human motions. Unlike existing motion datasets that rely on broad action labels or generic captions, MotionScript provides fine-grained, structured descriptions that capture the full complexity of human movement including expressive actions (e.g., emotions, stylistic walking) and interactions beyond standard motion capture datasets. MotionScript serves as both a descriptive tool and a training resource for text-to-motion models, enabling the synthesis of highly realistic and diverse human motions from text. By augmenting motion datasets with MotionScript captions, we demonstrate significant improvements in out-of-distribution motion generation, allowing large language models (LLMs) to generate motions that extend beyond existing data. Additionally, MotionScript opens new applications in animation, virtual human simulation, and robotics, providing an interpretable bridge between intuitive descriptions and motion synthesis. To the best of our knowledge, this is the first attempt to systematically translate 3D motion into structured natural language without requiring training data.

Related papers

• Animating the Uncaptured: Humanoid Mesh Animation with Video Diffusion Models [71.78723353724493]
  Animation of humanoid characters is essential in various graphics applications. We propose an approach to synthesize 4D animated sequences of input static 3D humanoid meshes.
  arXiv  Detail & Related papers  (2025-03-20T10:00:22Z)
• Motion-2-to-3: Leveraging 2D Motion Data to Boost 3D Motion Generation [43.915871360698546]
  2D human videos offer a vast and accessible source of motion data, covering a wider range of styles and activities. We introduce a novel framework that disentangles local joint motion from global movements, enabling efficient learning of local motion priors from 2D data. Our method efficiently utilizes 2D data, supporting realistic 3D human motion generation and broadening the range of motion types it supports.
  arXiv  Detail & Related papers  (2024-12-17T17:34:52Z)
• Motion Prompting: Controlling Video Generation with Motion Trajectories [57.049252242807874]
  We train a video generation model conditioned on sparse or dense video trajectories. We translate high-level user requests into detailed, semi-dense motion prompts. We demonstrate our approach through various applications, including camera and object motion control, "interacting" with an image, motion transfer, and image editing.
  arXiv  Detail & Related papers  (2024-12-03T18:59:56Z)
• LocoMotion: Learning Motion-Focused Video-Language Representations [45.33444862034461]
  We propose LocoMotion to learn from motion-focused captions that describe the movement and temporal progression of local object motions. We achieve this by adding synthetic motions to videos and using the parameters of these motions to generate corresponding captions.
  arXiv  Detail & Related papers  (2024-10-15T19:33:57Z)
• Sitcom-Crafter: A Plot-Driven Human Motion Generation System in 3D Scenes [90.39860012099393]
  Sitcom-Crafter is a system for human motion generation in 3D space. Central to the function generation modules is our novel 3D scene-aware human-human interaction module. Augmentation modules encompass plot comprehension for command generation, motion synchronization for seamless integration of different motion types.
  arXiv  Detail & Related papers  (2024-10-14T17:56:19Z)
• Generating Human Interaction Motions in Scenes with Text Control [66.74298145999909]
  We present TeSMo, a method for text-controlled scene-aware motion generation based on denoising diffusion models. Our approach begins with pre-training a scene-agnostic text-to-motion diffusion model. To facilitate training, we embed annotated navigation and interaction motions within scenes.
  arXiv  Detail & Related papers  (2024-04-16T16:04:38Z)
• Motion Generation from Fine-grained Textual Descriptions [29.033358642532722]
  We build a large-scale language-motion dataset specializing in fine-grained textual descriptions, FineHumanML3D. We design a new text2motion model, FineMotionDiffuse, making full use of fine-grained textual information. Our evaluation shows that FineMotionDiffuse trained on FineHumanML3D improves FID by a large margin of 0.38, compared with competitive baselines.
  arXiv  Detail & Related papers  (2024-03-20T11:38:30Z)
• Plan, Posture and Go: Towards Open-World Text-to-Motion Generation [43.392549755386135]
  We present a divide-and-conquer framework named PRO-Motion. It consists of three modules as motion planner, posture-diffuser and go-diffuser. Pro-Motion can generate diverse and realistic motions from complex open-world prompts.
  arXiv  Detail & Related papers  (2023-12-22T17:02:45Z)
• LivePhoto: Real Image Animation with Text-guided Motion Control [51.31418077586208]
  This work presents a practical system, named LivePhoto, which allows users to animate an image of their interest with text descriptions. We first establish a strong baseline that helps a well-learned text-to-image generator (i.e., Stable Diffusion) take an image as a further input. We then equip the improved generator with a motion module for temporal modeling and propose a carefully designed training pipeline to better link texts and motions.
  arXiv  Detail & Related papers  (2023-12-05T17:59:52Z)
• MotionGPT: Human Motion as a Foreign Language [47.21648303282788]
  Human motion displays a semantic coupling akin to human language, often perceived as a form of body language. By fusing language data with large-scale motion models, motion-language pre-training can enhance the performance of motion-related tasks. We propose MotionGPT, a unified, versatile, and user-friendly motion-language model to handle multiple motion-relevant tasks.
  arXiv  Detail & Related papers  (2023-06-26T15:53:02Z)
• HUMANISE: Language-conditioned Human Motion Generation in 3D Scenes [54.61610144668777]
  We present a novel scene-and-language conditioned generative model that can produce 3D human motions in 3D scenes. Our experiments demonstrate that our model generates diverse and semantically consistent human motions in 3D scenes.
  arXiv  Detail & Related papers  (2022-10-18T10:14:11Z)
• MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model [35.32967411186489]
  MotionDiffuse is a diffusion model-based text-driven motion generation framework. It excels at modeling complicated data distribution and generating vivid motion sequences. It responds to fine-grained instructions on body parts, and arbitrary-length motion synthesis with time-varied text prompts.
  arXiv  Detail & Related papers  (2022-08-31T17:58:54Z)
• Language-Guided Face Animation by Recurrent StyleGAN-based Generator [87.56260982475564]
  We study a novel task, language-guided face animation, that aims to animate a static face image with the help of languages. We propose a recurrent motion generator to extract a series of semantic and motion information from the language and feed it along with visual information to a pre-trained StyleGAN to generate high-quality frames.
  arXiv  Detail & Related papers  (2022-08-11T02:57:30Z)
• Synthesis of Compositional Animations from Textual Descriptions [54.85920052559239]
  "How unstructured and complex can we make a sentence and still generate plausible movements from it?" "How can we animate 3D-characters from a movie script or move robots by simply telling them what we would like them to do?"
  arXiv  Detail & Related papers  (2021-03-26T18:23:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.