MonoOcc: Digging into Monocular Semantic Occupancy Prediction

• URL: http://arxiv.org/abs/2403.08766v1
• Date: Wed, 13 Mar 2024 17:59:04 GMT
• Title: MonoOcc: Digging into Monocular Semantic Occupancy Prediction
• Abstract summary: Monocular Semantic Occupancy Prediction aims to infer the complete 3D geometry and semantic information of scenes from only 2D images. Existing methods rely on a complex cascaded framework with relatively limited information to restore 3D scenes. Our method yields state-of-the-art performance on the camera-based SemanticKITTI Scene Completion benchmark.
• Score: 20.833710016477653
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Monocular Semantic Occupancy Prediction aims to infer the complete 3D geometry and semantic information of scenes from only 2D images. It has garnered significant attention, particularly due to its potential to enhance the 3D perception of autonomous vehicles. However, existing methods rely on a complex cascaded framework with relatively limited information to restore 3D scenes, including a dependency on supervision solely on the whole network's output, single-frame input, and the utilization of a small backbone. These challenges, in turn, hinder the optimization of the framework and yield inferior prediction results, particularly concerning smaller and long-tailed objects. To address these issues, we propose MonoOcc. In particular, we (i) improve the monocular occupancy prediction framework by proposing an auxiliary semantic loss as supervision to the shallow layers of the framework and an image-conditioned cross-attention module to refine voxel features with visual clues, and (ii) employ a distillation module that transfers temporal information and richer knowledge from a larger image backbone to the monocular semantic occupancy prediction framework with low cost of hardware. With these advantages, our method yields state-of-the-art performance on the camera-based SemanticKITTI Scene Completion benchmark. Codes and models can be accessed at https://github.com/ucaszyp/MonoOcc

Related papers

• UnsOcc: 3D Semantic Occupancy Prediction in Unstructured Scene via Rendering Fusion [2.4196046716427393]
  UnsOcc is a multi-modal 3D semantic occupancy prediction framework that improves robustness in unstructured environments.<n>At its core, we introduce a rendering-based fusion module, RenderFusion, which enhances cross-modal feature alignment.<n>Our method significantly outperforms existing state-of-the-art approaches.
  arXiv  Detail & Related papers  (2026-06-02T12:50:14Z)
• HENet++: Hybrid Encoding and Multi-task Learning for 3D Perception and End-to-end Autonomous Driving [47.368036613468455]
  We present the HENet and HENet++ framework for multi-task 3D perception and end-to-end autonomous driving.<n>Specifically, we propose a hybrid image encoding network that uses a large image encoder for short-term frames and a small one for long-term frames.<n>Our framework simultaneously extracts both dense and sparse features, providing more suitable representations for different tasks, reducing cumulative errors, and delivering more comprehensive information to the planning module.
  arXiv  Detail & Related papers  (2025-11-10T13:49:59Z)
• Complementary Information Guided Occupancy Prediction via Multi-Level Representation Fusion [73.11061598576798]
  Camera-based occupancy prediction is a mainstream approach for 3D perception in autonomous driving.<n>textbfCIGOcc is a two-stage occupancy prediction framework based on multi-level representation fusion.<n>textbfCIGOcc extracts segmentation, graphics, and depth features from an input image and introduces a deformable multi-level fusion mechanism.
  arXiv  Detail & Related papers  (2025-10-15T06:37:33Z)
• DGOcc: Depth-aware Global Query-based Network for Monocular 3D Occupancy Prediction [17.38916914453357]
  Predicting the 3D occupancy of large-scale outdoor scenes from 2D images is ill-posed and resource-intensive. We present textbfDGOcc, a textbfGlobal query-based network for monocular 3D textbfOccupancy prediction. The proposed method achieves the best performance on monocular semantic occupancy prediction while reducing GPU and time overhead.
  arXiv  Detail & Related papers  (2025-04-10T07:44:55Z)
• MESC-3D:Mining Effective Semantic Cues for 3D Reconstruction from a Single Image [8.095737075287204]
  We propose a novel single-image 3D reconstruction method called Mining Effective Semantic Cues for 3D Reconstruction from a Single Image (MESC-3D) Specifically, we design an Effective Semantic Mining Module to establish connections between point clouds and image semantic attributes. This module incorporates semantic understanding of spatial structures, enabling the model to interpret and reconstruct 3D objects with greater accuracy and realism.
  arXiv  Detail & Related papers  (2025-02-28T09:02:15Z)
• Hierarchical Temporal Context Learning for Camera-based Semantic Scene Completion [57.232688209606515]
  We present HTCL, a novel Temporal Temporal Context Learning paradigm for improving camera-based semantic scene completion. Our method ranks $1st$ on the Semantic KITTI benchmark and even surpasses LiDAR-based methods in terms of mIoU.
  arXiv  Detail & Related papers  (2024-07-02T09:11:17Z)
• DistillNeRF: Perceiving 3D Scenes from Single-Glance Images by Distilling Neural Fields and Foundation Model Features [65.8738034806085]
  DistillNeRF is a self-supervised learning framework for understanding 3D environments in autonomous driving scenes. Our method is a generalizable feedforward model that predicts a rich neural scene representation from sparse, single-frame multi-view camera inputs.
  arXiv  Detail & Related papers  (2024-06-17T21:15:13Z)
• Scaling Multi-Camera 3D Object Detection through Weak-to-Strong Eliciting [32.66151412557986]
  We present a weak-to-strong eliciting framework aimed at enhancing surround refinement while maintaining robust monocular perception. Our framework employs weakly tuned experts trained on distinct subsets, and each is inherently biased toward specific camera configurations and scenarios. For MC3D-Det joint training, the elaborate dataset merge strategy is designed to solve the problem of inconsistent camera numbers and camera parameters.
  arXiv  Detail & Related papers  (2024-04-10T03:11:10Z)
• Rethinking Amodal Video Segmentation from Learning Supervised Signals with Object-centric Representation [47.39455910191075]
  Video amodal segmentation is a challenging task in computer vision. Recent studies have achieved promising performance by using motion flow to integrate information across frames under a self-supervised setting. This paper presents a rethinking to previous works. We particularly leverage the supervised signals with object-centric representation.
  arXiv  Detail & Related papers  (2023-09-23T04:12:02Z)
• Exploring Optical-Flow-Guided Motion and Detection-Based Appearance for Temporal Sentence Grounding [61.57847727651068]
  Temporal sentence grounding aims to localize a target segment in an untrimmed video semantically according to a given sentence query. Most previous works focus on learning frame-level features of each whole frame in the entire video, and directly match them with the textual information. We propose a novel Motion- and Appearance-guided 3D Semantic Reasoning Network (MA3SRN), which incorporates optical-flow-guided motion-aware, detection-based appearance-aware, and 3D-aware object-level features.
  arXiv  Detail & Related papers  (2022-03-06T13:57:09Z)
• SGM3D: Stereo Guided Monocular 3D Object Detection [62.11858392862551]
  We propose a stereo-guided monocular 3D object detection network, termed SGM3D. We exploit robust 3D features extracted from stereo images to enhance the features learned from the monocular image. Our method can be integrated into many other monocular approaches to boost performance without introducing any extra computational cost.
  arXiv  Detail & Related papers  (2021-12-03T13:57:14Z)
• Self-Supervised Multi-Frame Monocular Scene Flow [61.588808225321735]
  We introduce a multi-frame monocular scene flow network based on self-supervised learning. We observe state-of-the-art accuracy among monocular scene flow methods based on self-supervised learning.
  arXiv  Detail & Related papers  (2021-05-05T17:49:55Z)
• SynDistNet: Self-Supervised Monocular Fisheye Camera Distance Estimation Synergized with Semantic Segmentation for Autonomous Driving [37.50089104051591]
  State-of-the-art self-supervised learning approaches for monocular depth estimation usually suffer from scale ambiguity. This paper introduces a novel multi-task learning strategy to improve self-supervised monocular distance estimation on fisheye and pinhole camera images.
  arXiv  Detail & Related papers  (2020-08-10T10:52:47Z)
• 3D Sketch-aware Semantic Scene Completion via Semi-supervised Structure Prior [50.73148041205675]
  The goal of the Semantic Scene Completion (SSC) task is to simultaneously predict a completed 3D voxel representation of volumetric occupancy and semantic labels of objects in the scene from a single-view observation. We propose to devise a new geometry-based strategy to embed depth information with low-resolution voxel representation. Our proposed geometric embedding works better than the depth feature learning from habitual SSC frameworks.
  arXiv  Detail & Related papers  (2020-03-31T09:33:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.