HUNTER: Unsupervised Human-centric 3D Detection via Transferring Knowledge from Synthetic Instances to Real Scenes

• URL: http://arxiv.org/abs/2403.02769v2
• Date: Fri, 15 Mar 2024 15:46:54 GMT
• Title: HUNTER: Unsupervised Human-centric 3D Detection via Transferring Knowledge from Synthetic Instances to Real Scenes
• Authors: Yichen Yao, Zimo Jiang, Yujing Sun, Zhencai Zhu, Xinge Zhu, Runnan Chen, Yuexin Ma,
• Abstract summary: We propose an unsupervised 3D detection method for human-centric scenarios by transferring the knowledge from synthetic human instances to real scenes. Remarkably, our method exhibits superior performance compared to current state-of-the-art techniques.
• Score: 21.2539366684941
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Human-centric 3D scene understanding has recently drawn increasing attention, driven by its critical impact on robotics. However, human-centric real-life scenarios are extremely diverse and complicated, and humans have intricate motions and interactions. With limited labeled data, supervised methods are difficult to generalize to general scenarios, hindering real-life applications. Mimicking human intelligence, we propose an unsupervised 3D detection method for human-centric scenarios by transferring the knowledge from synthetic human instances to real scenes. To bridge the gap between the distinct data representations and feature distributions of synthetic models and real point clouds, we introduce novel modules for effective instance-to-scene representation transfer and synthetic-to-real feature alignment. Remarkably, our method exhibits superior performance compared to current state-of-the-art techniques, achieving 87.8% improvement in mAP and closely approaching the performance of fully supervised methods (62.15 mAP vs. 69.02 mAP) on HuCenLife Dataset.

Related papers

• Human-Aware 3D Scene Generation with Spatially-constrained Diffusion Models [16.259040755335885]
  Previous auto-regression-based 3D scene generation methods have struggled to accurately capture the joint distribution of multiple objects and input humans. We introduce two spatial collision guidance mechanisms: human-object collision avoidance and object-room boundary constraints. Our framework can generate more natural and plausible 3D scenes with precise human-scene interactions.
  arXiv  Detail & Related papers  (2024-06-26T08:18:39Z)
• Enhancing Generalizability of Representation Learning for Data-Efficient 3D Scene Understanding [50.448520056844885]
  We propose a generative Bayesian network to produce diverse synthetic scenes with real-world patterns. A series of experiments robustly display our method's consistent superiority over existing state-of-the-art pre-training approaches.
  arXiv  Detail & Related papers  (2024-06-17T07:43:53Z)
• Multimodal Sense-Informed Prediction of 3D Human Motions [16.71099574742631]
  This work introduces a novel multi-modal sense-informed motion prediction approach, which conditions high-fidelity generation on two modal information. The gaze information is regarded as the human intention, and combined with both motion and scene features, we construct a ternary intention-aware attention to supervise the generation. On two real-world benchmarks, the proposed method achieves state-of-the-art performance both in 3D human pose and trajectory prediction.
  arXiv  Detail & Related papers  (2024-05-05T12:38:10Z)
• Scaling Up Dynamic Human-Scene Interaction Modeling [58.032368564071895]
  TRUMANS is the most comprehensive motion-captured HSI dataset currently available. It intricately captures whole-body human motions and part-level object dynamics. We devise a diffusion-based autoregressive model that efficiently generates HSI sequences of any length.
  arXiv  Detail & Related papers  (2024-03-13T15:45:04Z)
• Human-centric Scene Understanding for 3D Large-scale Scenarios [52.12727427303162]
  We present a large-scale multi-modal dataset for human-centric scene understanding, dubbed HuCenLife. Our HuCenLife can benefit many 3D perception tasks, such as segmentation, detection, action recognition, etc.
  arXiv  Detail & Related papers  (2023-07-26T08:40:46Z)
• HULC: 3D Human Motion Capture with Pose Manifold Sampling and Dense Contact Guidance [82.09463058198546]
  Marker-less monocular 3D human motion capture (MoCap) with scene interactions is a challenging research topic relevant for extended reality, robotics and virtual avatar generation. We propose HULC, a new approach for 3D human MoCap which is aware of the scene geometry.
  arXiv  Detail & Related papers  (2022-05-11T17:59:31Z)
• HSPACE: Synthetic Parametric Humans Animated in Complex Environments [67.8628917474705]
  We build a large-scale photo-realistic dataset, Human-SPACE, of animated humans placed in complex indoor and outdoor environments. We combine a hundred diverse individuals of varying ages, gender, proportions, and ethnicity, with hundreds of motions and scenes, in order to generate an initial dataset of over 1 million frames. Assets are generated automatically, at scale, and are compatible with existing real time rendering and game engines.
  arXiv  Detail & Related papers  (2021-12-23T22:27:55Z)
• PLACE: Proximity Learning of Articulation and Contact in 3D Environments [70.50782687884839]
  We propose a novel interaction generation method, named PLACE, which explicitly models the proximity between the human body and the 3D scene around it. Our perceptual study shows that PLACE significantly improves the state-of-the-art method, approaching the realism of real human-scene interaction.
  arXiv  Detail & Related papers  (2020-08-12T21:00:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.