Learning 3D Scene Analogies with Neural Contextual Scene Maps

• URL: http://arxiv.org/abs/2503.15897v1
• Date: Thu, 20 Mar 2025 06:49:33 GMT
• Title: Learning 3D Scene Analogies with Neural Contextual Scene Maps
• Authors: Junho Kim, Gwangtak Bae, Eun Sun Lee, Young Min Kim,
• Abstract summary: We propose teaching machines to identify relational commonalities in 3D spaces.<n>Instead of focusing on point-wise or object-wise representations, we introduce 3D scene analogies.
• Score: 17.545689536966265
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Understanding scene contexts is crucial for machines to perform tasks and adapt prior knowledge in unseen or noisy 3D environments. As data-driven learning is intractable to comprehensively encapsulate diverse ranges of layouts and open spaces, we propose teaching machines to identify relational commonalities in 3D spaces. Instead of focusing on point-wise or object-wise representations, we introduce 3D scene analogies, which are smooth maps between 3D scene regions that align spatial relationships. Unlike well-studied single instance-level maps, these scene-level maps smoothly link large scene regions, potentially enabling unique applications in trajectory transfer in AR/VR, long demonstration transfer for imitation learning, and context-aware object rearrangement. To find 3D scene analogies, we propose neural contextual scene maps, which extract descriptor fields summarizing semantic and geometric contexts, and holistically align them in a coarse-to-fine manner for map estimation. This approach reduces reliance on individual feature points, making it robust to input noise or shape variations. Experiments demonstrate the effectiveness of our approach in identifying scene analogies and transferring trajectories or object placements in diverse indoor scenes, indicating its potential for robotics and AR/VR applications.

Related papers

• SliceOcc: Indoor 3D Semantic Occupancy Prediction with Vertical Slice Representation [50.420711084672966]
  We present SliceOcc, an RGB camera-based model specifically tailored for indoor 3D semantic occupancy prediction. Experimental results on the EmbodiedScan dataset demonstrate that SliceOcc achieves a mIoU of 15.45% across 81 indoor categories.
  arXiv  Detail & Related papers  (2025-01-28T03:41:24Z)
• Self-supervised Learning of Neural Implicit Feature Fields for Camera Pose Refinement [32.335953514942474]
  This paper proposes to jointly learn the scene representation along with a 3D dense feature field and a 2D feature extractor. We learn the underlying geometry of the scene with an implicit field through volumetric rendering and design our feature field to leverage intermediate geometric information encoded in the implicit field. Visual localization is then achieved by aligning the image-based features and the rendered volumetric features.
  arXiv  Detail & Related papers  (2024-06-12T17:51:53Z)
• CommonScenes: Generating Commonsense 3D Indoor Scenes with Scene Graph Diffusion [83.30168660888913]
  We present CommonScenes, a fully generative model that converts scene graphs into corresponding controllable 3D scenes. Our pipeline consists of two branches, one predicting the overall scene layout via a variational auto-encoder and the other generating compatible shapes. The generated scenes can be manipulated by editing the input scene graph and sampling the noise in the diffusion model.
  arXiv  Detail & Related papers  (2023-05-25T17:39:13Z)
• Generating Visual Spatial Description via Holistic 3D Scene Understanding [88.99773815159345]
  Visual spatial description (VSD) aims to generate texts that describe the spatial relations of the given objects within images. With an external 3D scene extractor, we obtain the 3D objects and scene features for input images. We construct a target object-centered 3D spatial scene graph (Go3D-S2G), such that we model the spatial semantics of target objects within the holistic 3D scenes.
  arXiv  Detail & Related papers  (2023-05-19T15:53:56Z)
• Neural Implicit Dense Semantic SLAM [83.04331351572277]
  We propose a novel RGBD vSLAM algorithm that learns a memory-efficient, dense 3D geometry, and semantic segmentation of an indoor scene in an online manner. Our pipeline combines classical 3D vision-based tracking and loop closing with neural fields-based mapping. Our proposed algorithm can greatly enhance scene perception and assist with a range of robot control problems.
  arXiv  Detail & Related papers  (2023-04-27T23:03:52Z)
• OpenScene: 3D Scene Understanding with Open Vocabularies [73.1411930820683]
  Traditional 3D scene understanding approaches rely on labeled 3D datasets to train a model for a single task with supervision. We propose OpenScene, an alternative approach where a model predicts dense features for 3D scene points that are co-embedded with text and image pixels in CLIP feature space. This zero-shot approach enables task-agnostic training and open-vocabulary queries.
  arXiv  Detail & Related papers  (2022-11-28T18:58:36Z)
• SinGRAV: Learning a Generative Radiance Volume from a Single Natural Scene [42.24260323525382]
  We present a 3D generative model for general natural scenes. Lacking necessary volumes of 3D data characterizing the target scene, we propose to learn from a single scene. We exploit a multi-scale convolutional network, which possesses the spatial locality bias in nature, to learn from the statistics of local regions at multiple scales within a single scene.
  arXiv  Detail & Related papers  (2022-10-03T19:38:14Z)
• Neural Groundplans: Persistent Neural Scene Representations from a Single Image [90.04272671464238]
  We present a method to map 2D image observations of a scene to a persistent 3D scene representation. We propose conditional neural groundplans as persistent and memory-efficient scene representations.
  arXiv  Detail & Related papers  (2022-07-22T17:41:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.