Human-Aware Object Placement for Visual Environment Reconstruction

• URL: http://arxiv.org/abs/2203.03609v1
• Date: Mon, 7 Mar 2022 18:59:02 GMT
• Title: Human-Aware Object Placement for Visual Environment Reconstruction
• Authors: Hongwei Yi and Chun-Hao P. Huang and Dimitrios Tzionas and Muhammed Kocabas and Mohamed Hassan and Siyu Tang and Justus Thies and Michael J. Black
• Abstract summary: We show that human-scene interactions can be leveraged to improve the 3D reconstruction of a scene from a monocular RGB video. Our key idea is that, as a person moves through a scene and interacts with it, we accumulate HSIs across multiple input images. We show that our scene reconstruction can be used to refine the initial 3D human pose and shape estimation.
• Score: 63.14733166375534
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Humans are in constant contact with the world as they move through it and interact with it. This contact is a vital source of information for understanding 3D humans, 3D scenes, and the interactions between them. In fact, we demonstrate that these human-scene interactions (HSIs) can be leveraged to improve the 3D reconstruction of a scene from a monocular RGB video. Our key idea is that, as a person moves through a scene and interacts with it, we accumulate HSIs across multiple input images, and optimize the 3D scene to reconstruct a consistent, physically plausible and functional 3D scene layout. Our optimization-based approach exploits three types of HSI constraints: (1) humans that move in a scene are occluded or occlude objects, thus, defining the depth ordering of the objects, (2) humans move through free space and do not interpenetrate objects, (3) when humans and objects are in contact, the contact surfaces occupy the same place in space. Using these constraints in an optimization formulation across all observations, we significantly improve the 3D scene layout reconstruction. Furthermore, we show that our scene reconstruction can be used to refine the initial 3D human pose and shape (HPS) estimation. We evaluate the 3D scene layout reconstruction and HPS estimation qualitatively and quantitatively using the PROX and PiGraphs datasets. The code and data are available for research purposes at https://mover.is.tue.mpg.de/.

Related papers

• Joint Reconstruction of 3D Human and Object via Contact-Based Refinement Transformer [58.98785899556135]
  We present a novel joint 3D human-object reconstruction method (CONTHO) that effectively exploits contact information between humans and objects. There are two core designs in our system: 1) 3D-guided contact estimation and 2) contact-based 3D human and object refinement.
  arXiv  Detail & Related papers  (2024-04-07T06:01:49Z)
• DECO: Dense Estimation of 3D Human-Scene Contact In The Wild [54.44345845842109]
  We train a novel 3D contact detector that uses both body-part-driven and scene-context-driven attention to estimate contact on the SMPL body. We significantly outperform existing SOTA methods across all benchmarks. We also show qualitatively that DECO generalizes well to diverse and challenging real-world human interactions in natural images.
  arXiv  Detail & Related papers  (2023-09-26T21:21:07Z)
• Physically Plausible 3D Human-Scene Reconstruction from Monocular RGB Image using an Adversarial Learning Approach [26.827712050966]
  A key challenge in holistic 3D human-scene reconstruction is to generate a physically plausible 3D scene from a single monocular RGB image. This paper proposes using an implicit feature representation of the scene elements to distinguish a physically plausible alignment of humans and objects. Unlike the existing inference-time optimization-based approaches, we use this adversarially trained model to produce a per-frame 3D reconstruction of the scene.
  arXiv  Detail & Related papers  (2023-07-27T01:07:15Z)
• Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors [71.29186299435423]
  We introduce (HPS) Human POSEitioning System, a method to recover the full 3D pose of a human registered with a 3D scan of the surrounding environment. We show that our optimization-based integration exploits the benefits of the two, resulting in pose accuracy free of drift. HPS could be used for VR/AR applications where humans interact with the scene without requiring direct line of sight with an external camera.
  arXiv  Detail & Related papers  (2021-03-31T17:58:31Z)
• Populating 3D Scenes by Learning Human-Scene Interaction [47.42049393299]
  We learn how humans interact with scenes and leverage this to enable virtual characters to do the same. The representation of interaction is body-centric, which enables it to generalize to new scenes. We show that POSA's learned representation of body-scene interaction supports monocular human pose estimation.
  arXiv  Detail & Related papers  (2020-12-21T18:57:55Z)
• PLACE: Proximity Learning of Articulation and Contact in 3D Environments [70.50782687884839]
  We propose a novel interaction generation method, named PLACE, which explicitly models the proximity between the human body and the 3D scene around it. Our perceptual study shows that PLACE significantly improves the state-of-the-art method, approaching the realism of real human-scene interaction.
  arXiv  Detail & Related papers  (2020-08-12T21:00:10Z)
• Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wild [96.08358373137438]
  We present a method that infers spatial arrangements and shapes of humans and objects in a globally consistent 3D scene. Our method runs on datasets without any scene- or object-level 3D supervision.
  arXiv  Detail & Related papers  (2020-07-30T17:59:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.