SINC: Spatial Composition of 3D Human Motions for Simultaneous Action Generation

• URL: http://arxiv.org/abs/2304.10417v3
• Date: Tue, 26 Mar 2024 13:16:02 GMT
• Title: SINC: Spatial Composition of 3D Human Motions for Simultaneous Action Generation
• Authors: Nikos Athanasiou, Mathis Petrovich, Michael J. Black, Gül Varol,
• Abstract summary: Our goal is to synthesize 3D human motions given textual inputs describing simultaneous actions. We refer to generating such simultaneous movements as performing'spatial compositions'
• Score: 58.25766404147109
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Our goal is to synthesize 3D human motions given textual inputs describing simultaneous actions, for example 'waving hand' while 'walking' at the same time. We refer to generating such simultaneous movements as performing 'spatial compositions'. In contrast to temporal compositions that seek to transition from one action to another, spatial compositing requires understanding which body parts are involved in which action, to be able to move them simultaneously. Motivated by the observation that the correspondence between actions and body parts is encoded in powerful language models, we extract this knowledge by prompting GPT-3 with text such as "what are the body parts involved in the action <action name>?", while also providing the parts list and few-shot examples. Given this action-part mapping, we combine body parts from two motions together and establish the first automated method to spatially compose two actions. However, training data with compositional actions is always limited by the combinatorics. Hence, we further create synthetic data with this approach, and use it to train a new state-of-the-art text-to-motion generation model, called SINC ("SImultaneous actioN Compositions for 3D human motions"). In our experiments, that training with such GPT-guided synthetic data improves spatial composition generation over baselines. Our code is publicly available at https://sinc.is.tue.mpg.de/.

Related papers

• Contact-aware Human Motion Generation from Textual Descriptions [57.871692507044344]
  We create a novel dataset named RICH-CAT, representing Contact-Aware Texts'' We propose a novel approach named CATMO for text-driven interactive human motion synthesis. Our experiments demonstrate the superior performance of our approach compared to existing text-to-motion methods.
  arXiv  Detail & Related papers  (2024-03-23T04:08:39Z)
• Motion Generation from Fine-grained Textual Descriptions [29.033358642532722]
  We build a large-scale language-motion dataset specializing in fine-grained textual descriptions, FineHumanML3D. We design a new text2motion model, FineMotionDiffuse, making full use of fine-grained textual information. Our evaluation shows that FineMotionDiffuse trained on FineHumanML3D improves FID by a large margin of 0.38, compared with competitive baselines.
  arXiv  Detail & Related papers  (2024-03-20T11:38:30Z)
• GRIP: Generating Interaction Poses Using Spatial Cues and Latent Consistency [57.9920824261925]
  Hands are dexterous and highly versatile manipulators that are central to how humans interact with objects and their environment. modeling realistic hand-object interactions is critical for applications in computer graphics, computer vision, and mixed reality. GRIP is a learning-based method that takes as input the 3D motion of the body and the object, and synthesizes realistic motion for both hands before, during, and after object interaction.
  arXiv  Detail & Related papers  (2023-08-22T17:59:51Z)
• IMoS: Intent-Driven Full-Body Motion Synthesis for Human-Object Interactions [69.95820880360345]
  We present the first framework to synthesize the full-body motion of virtual human characters with 3D objects placed within their reach. Our system takes as input textual instructions specifying the objects and the associated intentions of the virtual characters. We show that our synthesized full-body motions appear more realistic to the participants in more than 80% of scenarios.
  arXiv  Detail & Related papers  (2022-12-14T23:59:24Z)
• TEACH: Temporal Action Composition for 3D Humans [50.97135662063117]
  Given a series of natural language descriptions, our task is to generate 3D human motions that correspond semantically to the text. In particular, our goal is to enable the synthesis of a series of actions, which we refer to as temporal action composition.
  arXiv  Detail & Related papers  (2022-09-09T00:33:40Z)
• Synthesis of Compositional Animations from Textual Descriptions [54.85920052559239]
  "How unstructured and complex can we make a sentence and still generate plausible movements from it?" "How can we animate 3D-characters from a movie script or move robots by simply telling them what we would like them to do?"
  arXiv  Detail & Related papers  (2021-03-26T18:23:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.