VisHall3D: Monocular Semantic Scene Completion from Reconstructing the Visible Regions to Hallucinating the Invisible Regions

• URL: http://arxiv.org/abs/2507.19188v1
• Date: Fri, 25 Jul 2025 11:57:18 GMT
• Title: VisHall3D: Monocular Semantic Scene Completion from Reconstructing the Visible Regions to Hallucinating the Invisible Regions
• Authors: Haoang Lu, Yuanqi Su, Xiaoning Zhang, Longjun Gao, Yu Xue, Le Wang,
• Abstract summary: VisHall3D decomposes the scene completion task into two stages: reconstructing the visible regions (vision) and inferring the invisible regions (hallucination)<n> VisHall3D achieves state-of-the-art performance, outperforming previous methods by a significant margin.
• Score: 8.379218197127216
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper introduces VisHall3D, a novel two-stage framework for monocular semantic scene completion that aims to address the issues of feature entanglement and geometric inconsistency prevalent in existing methods. VisHall3D decomposes the scene completion task into two stages: reconstructing the visible regions (vision) and inferring the invisible regions (hallucination). In the first stage, VisFrontierNet, a visibility-aware projection module, is introduced to accurately trace the visual frontier while preserving fine-grained details. In the second stage, OcclusionMAE, a hallucination network, is employed to generate plausible geometries for the invisible regions using a noise injection mechanism. By decoupling scene completion into these two distinct stages, VisHall3D effectively mitigates feature entanglement and geometric inconsistency, leading to significantly improved reconstruction quality. The effectiveness of VisHall3D is validated through extensive experiments on two challenging benchmarks: SemanticKITTI and SSCBench-KITTI-360. VisHall3D achieves state-of-the-art performance, outperforming previous methods by a significant margin and paves the way for more accurate and reliable scene understanding in autonomous driving and other applications.

Related papers

• VOIC: Visible-Occluded Decoupling for Monocular 3D Semantic Scene Completion [6.144392125326462]
  Camera-based 3D Semantic Scene Completion is a critical task for autonomous driving and robotic scene understanding.<n>Existing methods typically focus on end-to-end 2D-to-3D feature lifting and voxel completion.<n>We propose a novel dual-decoder framework that explicitly decouples SSC into visible-region semantic perception and occluded-region scene completion.
  arXiv  Detail & Related papers  (2025-12-22T02:05:45Z)
• HD$^2$-SSC: High-Dimension High-Density Semantic Scene Completion for Autonomous Driving [52.959716866316604]
  Camera-based 3D semantic scene completion (SSC) plays a crucial role in autonomous driving.<n>Existing SSC methods suffer from the inherent input-output dimension gap and annotation-reality density gap.<n>We propose a corresponding High- Dimension High-Density Semantic Scene Completion framework with expanded pixel semantics and refined voxel occupancies.
  arXiv  Detail & Related papers  (2025-11-11T07:24:35Z)
• Where, Not What: Compelling Video LLMs to Learn Geometric Causality for 3D-Grounding [0.8883733362171032]
  We propose a novel training framework called What-Where Representation Re-Forming (W2R2) to tackle this issue.<n>Our approach fundamentally reshapes the model's internal space by designating 2D features as semantic beacons for "What" identification and 3D features as spatial anchors for "Where" localization.<n>Experiments conducted on ScanRefer and ScanQA demonstrate the effectiveness of W2R2, with significant gains in localization accuracy and robustness.
  arXiv  Detail & Related papers  (2025-10-19T22:40:18Z)
• G4Splat: Geometry-Guided Gaussian Splatting with Generative Prior [53.762256749551284]
  We identify accurate geometry as the fundamental prerequisite for effectively exploiting generative models to enhance 3D scene reconstruction.<n>We incorporate this geometry guidance throughout the generative pipeline to improve visibility mask estimation, guide novel view selection, and enhance multi-view consistency when inpainting with video diffusion models.<n>Our method naturally supports single-view inputs and unposed videos, with strong generalizability in both indoor and outdoor scenarios.
  arXiv  Detail & Related papers  (2025-10-14T03:06:28Z)
• One Step Closer: Creating the Future to Boost Monocular Semantic Scene Completion [3.664655957801223]
  In real-world traffic scenarios, a significant portion of a visual 3D scene remains occluded or outside the camera's field of view.<n>We propose Creating the Future SSC, a novel temporal SSC framework that leverages pseudo-future frame prediction to expand the model's effective perceptual range.<n>Our approach combines poses and depths to establish accurate 3D correspondences, enabling geometrically-consistent fusion of past, present, and predicted future frames in 3D space.
  arXiv  Detail & Related papers  (2025-07-18T10:24:58Z)
• Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigation [54.04601077224252]
  Embodied scene understanding requires not only comprehending visual-spatial information but also determining where to explore next in the 3D physical world.<n>underlinetextbf3D vision-language learning enables embodied agents to effectively explore and understand their environment.<n>model's versatility enables navigation using diverse input modalities, including categories, language descriptions, and reference images.
  arXiv  Detail & Related papers  (2025-07-05T14:15:52Z)
• Bridging 3D Anomaly Localization and Repair via High-Quality Continuous Geometric Representation [1.4763103835215192]
  New framework integrates 3D anomaly detection and repair by learning a continuous, pose-invariant shape representation.<n>Experiments on Real3D-AD and Anomaly-ShapeNet demonstrate state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-05-30T10:11:49Z)
• GSFF-SLAM: 3D Semantic Gaussian Splatting SLAM via Feature Field [17.57215792490409]
  GSFF-SLAM is a novel dense semantic SLAM system based on 3D Gaussian Splatting.<n>Our method supports semantic reconstruction using various forms of 2D priors, particularly sparse and noisy signals.<n>When utilizing 2D ground truth priors, GSFF-SLAM achieves state-of-the-art semantic segmentation performance with 95.03% mIoU.
  arXiv  Detail & Related papers  (2025-04-28T01:21:35Z)
• econSG: Efficient and Multi-view Consistent Open-Vocabulary 3D Semantic Gaussians [56.85804719947]
  We propose econSG for open-vocabulary semantic segmentation with 3DGS.<n>Our econSG shows state-of-the-art performance on four benchmark datasets compared to the existing methods.
  arXiv  Detail & Related papers  (2025-04-08T13:12:31Z)
• OccScene: Semantic Occupancy-based Cross-task Mutual Learning for 3D Scene Generation [84.32038395034868]
  OccScene integrates fine-grained 3D perception and high-quality generation in a unified framework.<n>OccScene generates new and consistent 3D realistic scenes only depending on text prompts.<n>Experiments show that OccScene achieves realistic 3D scene generation in broad indoor and outdoor scenarios.
  arXiv  Detail & Related papers  (2024-12-15T13:26:51Z)
• Volumetric Environment Representation for Vision-Language Navigation [66.04379819772764]
  Vision-language navigation (VLN) requires an agent to navigate through a 3D environment based on visual observations and natural language instructions. We introduce a Volumetric Environment Representation (VER), which voxelizes the physical world into structured 3D cells. VER predicts 3D occupancy, 3D room layout, and 3D bounding boxes jointly.
  arXiv  Detail & Related papers  (2024-03-21T06:14:46Z)
• RadOcc: Learning Cross-Modality Occupancy Knowledge through Rendering Assisted Distillation [50.35403070279804]
  3D occupancy prediction is an emerging task that aims to estimate the occupancy states and semantics of 3D scenes using multi-view images. We propose RadOcc, a Rendering assisted distillation paradigm for 3D Occupancy prediction.
  arXiv  Detail & Related papers  (2023-12-19T03:39:56Z)
• ALSTER: A Local Spatio-Temporal Expert for Online 3D Semantic Reconstruction [62.599588577671796]
  We propose an online 3D semantic segmentation method that incrementally reconstructs a 3D semantic map from a stream of RGB-D frames. Unlike offline methods, ours is directly applicable to scenarios with real-time constraints, such as robotics or mixed reality.
  arXiv  Detail & Related papers  (2023-11-29T20:30:18Z)
• Improving Neural Indoor Surface Reconstruction with Mask-Guided Adaptive Consistency Constraints [0.6749750044497732]
  We propose a two-stage training process, decouple view-dependent and view-independent colors, and leverage two novel consistency constraints to enhance detail reconstruction performance without requiring extra priors. Experiments on synthetic and real-world datasets show the capability of reducing the interference from prior estimation errors.
  arXiv  Detail & Related papers  (2023-09-18T13:05:23Z)
• Scene-Generalizable Interactive Segmentation of Radiance Fields [64.37093918762]
  We make the first attempt at Scene-Generalizable Interactive in Radiance Fields (SGISRF) We propose a novel SGISRF method, which can perform 3D object segmentation for novel (unseen) scenes represented by radiance fields, guided by only a few interactive user clicks in a given set of multi-view 2D images. Experiments on two real-world challenging benchmarks covering diverse scenes demonstrate 1) effectiveness and scene-generalizability of the proposed method, 2) favorable performance compared to classical method requiring scene-specific optimization.
  arXiv  Detail & Related papers  (2023-08-09T17:55:50Z)
• Cross-Dimensional Refined Learning for Real-Time 3D Visual Perception from Monocular Video [2.2299983745857896]
  We present a novel real-time capable learning method that jointly perceives a 3D scene's geometry structure and semantic labels. We propose an end-to-end cross-dimensional refinement neural network (CDRNet) to extract both 3D mesh and 3D semantic labeling in real time.
  arXiv  Detail & Related papers  (2023-03-16T11:53:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.