Jointly Understand Your Command and Intention:Reciprocal Co-Evolution between Scene-Aware 3D Human Motion Synthesis and Analysis

• URL: http://arxiv.org/abs/2503.00371v3
• Date: Thu, 20 Mar 2025 13:20:09 GMT
• Title: Jointly Understand Your Command and Intention:Reciprocal Co-Evolution between Scene-Aware 3D Human Motion Synthesis and Analysis
• Authors: Xuehao Gao, Yang Yang, Shaoyi Du, Guo-Jun Qi, Junwei Han,
• Abstract summary: Scene-aware text-to-human synthesis generates diverse indoor motion samples from the same textual description.<n>We propose a cascaded generation strategy that factorizes text-driven scene-specific human motion generation into three stages.<n>We jointly improve realistic human motion synthesis and robust human motion analysis in 3D scenes.
• Score: 80.50342609047091
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As two intimate reciprocal tasks, scene-aware human motion synthesis and analysis require a joint understanding between multiple modalities, including 3D body motions, 3D scenes, and textual descriptions. In this paper, we integrate these two paired processes into a Co-Evolving Synthesis-Analysis (CESA) pipeline and mutually benefit their learning. Specifically, scene-aware text-to-human synthesis generates diverse indoor motion samples from the same textual description to enrich human-scene interaction intra-class diversity, thus significantly benefiting training a robust human motion analysis system. Reciprocally, human motion analysis would enforce semantic scrutiny on each synthesized motion sample to ensure its semantic consistency with the given textual description, thus improving realistic motion synthesis. Considering that real-world indoor human motions are goal-oriented and path-guided, we propose a cascaded generation strategy that factorizes text-driven scene-specific human motion generation into three stages: goal inferring, path planning, and pose synthesizing. Coupling CESA with this powerful cascaded motion synthesis model, we jointly improve realistic human motion synthesis and robust human motion analysis in 3D scenes.

Related papers

• AMD:Anatomical Motion Diffusion with Interpretable Motion Decomposition and Fusion [11.689663297469945]
  We propose the Adaptable Motion Diffusion model. It exploits a Large Language Model (LLM) to parse the input text into a sequence of concise and interpretable anatomical scripts. We then devise a two-branch fusion scheme that balances the influence of the input text and the anatomical scripts on the inverse diffusion process.
  arXiv  Detail & Related papers  (2023-12-20T04:49:45Z)
• ReMoS: 3D Motion-Conditioned Reaction Synthesis for Two-Person Interactions [66.87211993793807]
  We present ReMoS, a denoising diffusion based model that synthesizes full body motion of a person in two person interaction scenario. We demonstrate ReMoS across challenging two person scenarios such as pair dancing, Ninjutsu, kickboxing, and acrobatics. We also contribute the ReMoCap dataset for two person interactions containing full body and finger motions.
  arXiv  Detail & Related papers  (2023-11-28T18:59:52Z)
• SINC: Spatial Composition of 3D Human Motions for Simultaneous Action Generation [58.25766404147109]
  Our goal is to synthesize 3D human motions given textual inputs describing simultaneous actions. We refer to generating such simultaneous movements as performing'spatial compositions'
  arXiv  Detail & Related papers  (2023-04-20T16:01:55Z)
• IMoS: Intent-Driven Full-Body Motion Synthesis for Human-Object Interactions [69.95820880360345]
  We present the first framework to synthesize the full-body motion of virtual human characters with 3D objects placed within their reach. Our system takes as input textual instructions specifying the objects and the associated intentions of the virtual characters. We show that our synthesized full-body motions appear more realistic to the participants in more than 80% of scenarios.
  arXiv  Detail & Related papers  (2022-12-14T23:59:24Z)
• Skeleton2Humanoid: Animating Simulated Characters for Physically-plausible Motion In-betweening [59.88594294676711]
  Modern deep learning based motion synthesis approaches barely consider the physical plausibility of synthesized motions. We propose a system Skeleton2Humanoid'' which performs physics-oriented motion correction at test time. Experiments on the challenging LaFAN1 dataset show our system can outperform prior methods significantly in terms of both physical plausibility and accuracy.
  arXiv  Detail & Related papers  (2022-10-09T16:15:34Z)
• Towards Diverse and Natural Scene-aware 3D Human Motion Synthesis [117.15586710830489]
  We focus on the problem of synthesizing diverse scene-aware human motions under the guidance of target action sequences. Based on this factorized scheme, a hierarchical framework is proposed, with each sub-module responsible for modeling one aspect. Experiment results show that the proposed framework remarkably outperforms previous methods in terms of diversity and naturalness.
  arXiv  Detail & Related papers  (2022-05-25T18:20:01Z)
• Scene-aware Generative Network for Human Motion Synthesis [125.21079898942347]
  We propose a new framework, with the interaction between the scene and the human motion taken into account. Considering the uncertainty of human motion, we formulate this task as a generative task. We derive a GAN based learning approach, with discriminators to enforce the compatibility between the human motion and the contextual scene.
  arXiv  Detail & Related papers  (2021-05-31T09:05:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.