Locomotion-Action-Manipulation: Synthesizing Human-Scene Interactions in Complex 3D Environments

• URL: http://arxiv.org/abs/2301.02667v2
• Date: Fri, 8 Sep 2023 12:52:09 GMT
• Title: Locomotion-Action-Manipulation: Synthesizing Human-Scene Interactions in Complex 3D Environments
• Authors: Jiye Lee, Hanbyul Joo
• Abstract summary: We present LAMA, Locomotion-Action-MAnipulation, to synthesize natural and plausible long-term human movements in complex indoor environments. Unlike existing methods that require motion data "paired" with scanned 3D scenes for supervision, we formulate the problem as a test-time optimization by using human motion capture data only for synthesis.
• Score: 11.87902527509297
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Synthesizing interaction-involved human motions has been challenging due to the high complexity of 3D environments and the diversity of possible human behaviors within. We present LAMA, Locomotion-Action-MAnipulation, to synthesize natural and plausible long-term human movements in complex indoor environments. The key motivation of LAMA is to build a unified framework to encompass a series of everyday motions including locomotion, scene interaction, and object manipulation. Unlike existing methods that require motion data "paired" with scanned 3D scenes for supervision, we formulate the problem as a test-time optimization by using human motion capture data only for synthesis. LAMA leverages a reinforcement learning framework coupled with a motion matching algorithm for optimization, and further exploits a motion editing framework via manifold learning to cover possible variations in interaction and manipulation. Throughout extensive experiments, we demonstrate that LAMA outperforms previous approaches in synthesizing realistic motions in various challenging scenarios. Project page: https://jiyewise.github.io/projects/LAMA/ .

Related papers

• Sitcom-Crafter: A Plot-Driven Human Motion Generation System in 3D Scenes [83.55301458112672]
  Sitcom-Crafter is a system for human motion generation in 3D space. Central to the function generation modules is our novel 3D scene-aware human-human interaction module. Augmentation modules encompass plot comprehension for command generation, motion synchronization for seamless integration of different motion types.
  arXiv  Detail & Related papers  (2024-10-14T17:56:19Z)
• Scaling Up Dynamic Human-Scene Interaction Modeling [58.032368564071895]
  TRUMANS is the most comprehensive motion-captured HSI dataset currently available. It intricately captures whole-body human motions and part-level object dynamics. We devise a diffusion-based autoregressive model that efficiently generates HSI sequences of any length.
  arXiv  Detail & Related papers  (2024-03-13T15:45:04Z)
• InterControl: Zero-shot Human Interaction Generation by Controlling Every Joint [67.6297384588837]
  We introduce a novel controllable motion generation method, InterControl, to encourage the synthesized motions maintaining the desired distance between joint pairs. We demonstrate that the distance between joint pairs for human-wise interactions can be generated using an off-the-shelf Large Language Model.
  arXiv  Detail & Related papers  (2023-11-27T14:32:33Z)
• NIFTY: Neural Object Interaction Fields for Guided Human Motion Synthesis [21.650091018774972]
  We create a neural interaction field attached to a specific object, which outputs the distance to the valid interaction manifold given a human pose as input. This interaction field guides the sampling of an object-conditioned human motion diffusion model. We synthesize realistic motions for sitting and lifting with several objects, outperforming alternative approaches in terms of motion quality and successful action completion.
  arXiv  Detail & Related papers  (2023-07-14T17:59:38Z)
• Synthesizing Diverse Human Motions in 3D Indoor Scenes [16.948649870341782]
  We present a novel method for populating 3D indoor scenes with virtual humans that can navigate in the environment and interact with objects in a realistic manner. Existing approaches rely on training sequences that contain captured human motions and the 3D scenes they interact with. We propose a reinforcement learning-based approach that enables virtual humans to navigate in 3D scenes and interact with objects realistically and autonomously.
  arXiv  Detail & Related papers  (2023-05-21T09:22:24Z)
• IMoS: Intent-Driven Full-Body Motion Synthesis for Human-Object Interactions [69.95820880360345]
  We present the first framework to synthesize the full-body motion of virtual human characters with 3D objects placed within their reach. Our system takes as input textual instructions specifying the objects and the associated intentions of the virtual characters. We show that our synthesized full-body motions appear more realistic to the participants in more than 80% of scenarios.
  arXiv  Detail & Related papers  (2022-12-14T23:59:24Z)
• Compositional Human-Scene Interaction Synthesis with Semantic Control [16.93177243590465]
  We aim to synthesize humans interacting with a given 3D scene controlled by high-level semantic specifications. We design a novel transformer-based generative model, in which the articulated 3D human body surface points and 3D objects are jointly encoded. Inspired by the compositional nature of interactions that humans can simultaneously interact with multiple objects, we define interaction semantics as the composition of varying numbers of atomic action-object pairs.
  arXiv  Detail & Related papers  (2022-07-26T11:37:44Z)
• Towards Diverse and Natural Scene-aware 3D Human Motion Synthesis [117.15586710830489]
  We focus on the problem of synthesizing diverse scene-aware human motions under the guidance of target action sequences. Based on this factorized scheme, a hierarchical framework is proposed, with each sub-module responsible for modeling one aspect. Experiment results show that the proposed framework remarkably outperforms previous methods in terms of diversity and naturalness.
  arXiv  Detail & Related papers  (2022-05-25T18:20:01Z)
• iGibson, a Simulation Environment for Interactive Tasks in Large Realistic Scenes [54.04456391489063]
  iGibson is a novel simulation environment to develop robotic solutions for interactive tasks in large-scale realistic scenes. Our environment contains fifteen fully interactive home-sized scenes populated with rigid and articulated objects. iGibson features enable the generalization of navigation agents, and that the human-iGibson interface and integrated motion planners facilitate efficient imitation learning of simple human demonstrated behaviors.
  arXiv  Detail & Related papers  (2020-12-05T02:14:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.