RendBEV: Semantic Novel View Synthesis for Self-Supervised Bird's Eye View Segmentation

• URL: http://arxiv.org/abs/2502.14792v1
• Date: Thu, 20 Feb 2025 18:11:44 GMT
• Title: RendBEV: Semantic Novel View Synthesis for Self-Supervised Bird's Eye View Segmentation
• Authors: Henrique Piñeiro Monteagudo, Leonardo Taccari, Aurel Pjetri, Francesco Sambo, Samuele Salti,
• Abstract summary: We present RendBEV, a new method for the self-supervised training of Bird's Eye View semantic segmentation networks. Our method enables zero-shot BEV semantic segmentation, and already delivers competitive results.
• Score: 9.72227798086777
• License:
• Abstract: Bird's Eye View (BEV) semantic maps have recently garnered a lot of attention as a useful representation of the environment to tackle assisted and autonomous driving tasks. However, most of the existing work focuses on the fully supervised setting, training networks on large annotated datasets. In this work, we present RendBEV, a new method for the self-supervised training of BEV semantic segmentation networks, leveraging differentiable volumetric rendering to receive supervision from semantic perspective views computed by a 2D semantic segmentation model. Our method enables zero-shot BEV semantic segmentation, and already delivers competitive results in this challenging setting. When used as pretraining to then fine-tune on labeled BEV ground-truth, our method significantly boosts performance in low-annotation regimes, and sets a new state of the art when fine-tuning on all available labels.

Related papers

• VQ-Map: Bird's-Eye-View Map Layout Estimation in Tokenized Discrete Space via Vector Quantization [108.68014173017583]
  Bird's-eye-view (BEV) map layout estimation requires an accurate and full understanding of the semantics for the environmental elements around the ego car. We propose to utilize a generative model similar to the Vector Quantized-Variational AutoEncoder (VQ-VAE) to acquire prior knowledge for the high-level BEV semantics in the tokenized discrete space. Thanks to the obtained BEV tokens accompanied with a codebook embedding encapsulating the semantics for different BEV elements in the groundtruth maps, we are able to directly align the sparse backbone image features with the obtained BEV tokens
  arXiv  Detail & Related papers  (2024-11-03T16:09:47Z)
• BEVPose: Unveiling Scene Semantics through Pose-Guided Multi-Modal BEV Alignment [8.098296280937518]
  We present BEVPose, a framework that integrates BEV representations from camera and lidar data, using sensor pose as a guiding supervisory signal. By leveraging pose information, we align and fuse multi-modal sensory inputs, facilitating the learning of latent BEV embeddings that capture both geometric and semantic aspects of the environment.
  arXiv  Detail & Related papers  (2024-10-28T12:40:27Z)
• OE-BevSeg: An Object Informed and Environment Aware Multimodal Framework for Bird's-eye-view Vehicle Semantic Segmentation [57.2213693781672]
  Bird's-eye-view (BEV) semantic segmentation is becoming crucial in autonomous driving systems. We propose OE-BevSeg, an end-to-end multimodal framework that enhances BEV segmentation performance. Our approach achieves state-of-the-art results by a large margin on the nuScenes dataset for vehicle segmentation.
  arXiv  Detail & Related papers  (2024-07-18T03:48:22Z)
• LetsMap: Unsupervised Representation Learning for Semantic BEV Mapping [23.366388601110913]
  We propose the first unsupervised representation learning approach to generate semantic BEV maps from a monocular frontal view (FV) image in a label-efficient manner. Our approach pretrains the network to independently reason about scene geometry and scene semantics using two disjoint neural pathways in an unsupervised manner. We achieve label-free pretraining by exploiting spatial and temporal consistency of FV images to learn scene geometry while relying on a novel temporal masked autoencoder formulation to encode the scene representation.
  arXiv  Detail & Related papers  (2024-05-29T08:03:36Z)
• OccFeat: Self-supervised Occupancy Feature Prediction for Pretraining BEV Segmentation Networks [75.10231099007494]
  We introduce a self-supervised pretraining method, called OccFeat, for Bird's-Eye-View (BEV) segmentation networks. With OccFeat, we pretrain a BEV network via occupancy prediction and feature distillation tasks. Models pretrained with our method exhibit improved BEV semantic segmentation performance, particularly in low-data scenarios.
  arXiv  Detail & Related papers  (2024-04-22T09:43:03Z)
• Semi-Supervised Learning for Visual Bird's Eye View Semantic Segmentation [16.3996408206659]
  We present a novel semi-supervised framework for visual BEV semantic segmentation to boost performance by exploiting unlabeled images during the training. A consistency loss that makes full use of unlabeled data is then proposed to constrain the model on not only semantic prediction but also the BEV feature. Experiments on the nuScenes and Argoverse datasets show that our framework can effectively improve prediction accuracy.
  arXiv  Detail & Related papers  (2023-08-28T12:23:36Z)
• A Cross-Scale Hierarchical Transformer with Correspondence-Augmented Attention for inferring Bird's-Eye-View Semantic Segmentation [13.013635162859108]
  Inferring BEV semantic segmentation conditioned on multi-camera-view images is a popular scheme in the community as cheap devices and real-time processing. We propose a novel cross-scale hierarchical Transformer with correspondence-augmented attention for semantic segmentation inferring. Our method has state-of-the-art performance in inferring BEV semantic segmentation conditioned on multi-camera-view images.
  arXiv  Detail & Related papers  (2023-04-07T13:52:47Z)
• SkyEye: Self-Supervised Bird's-Eye-View Semantic Mapping Using Monocular Frontal View Images [26.34702432184092]
  We propose the first self-supervised approach for generating a Bird's-Eye-View (BEV) semantic map using a single monocular image from the frontal view (FV) In training, we overcome the need for BEV ground truth annotations by leveraging the more easily available FV semantic annotations of video sequences. Our approach performs on par with the state-of-the-art fully supervised methods and achieves competitive results using only 1% of direct supervision in the BEV.
  arXiv  Detail & Related papers  (2023-02-08T18:02:09Z)
• Delving into the Devils of Bird's-eye-view Perception: A Review, Evaluation and Recipe [115.31507979199564]
  Learning powerful representations in bird's-eye-view (BEV) for perception tasks is trending and drawing extensive attention both from industry and academia. As sensor configurations get more complex, integrating multi-source information from different sensors and representing features in a unified view come of vital importance. The core problems for BEV perception lie in (a) how to reconstruct the lost 3D information via view transformation from perspective view to BEV; (b) how to acquire ground truth annotations in BEV grid; and (d) how to adapt and generalize algorithms as sensor configurations vary across different scenarios.
  arXiv  Detail & Related papers  (2022-09-12T15:29:13Z)
• GitNet: Geometric Prior-based Transformation for Birds-Eye-View Segmentation [105.19949897812494]
  Birds-eye-view (BEV) semantic segmentation is critical for autonomous driving. We present a novel two-stage Geometry Prior-based Transformation framework named GitNet.
  arXiv  Detail & Related papers  (2022-04-16T06:46:45Z)
• NEAT: Neural Attention Fields for End-to-End Autonomous Driving [59.60483620730437]
  We present NEural ATtention fields (NEAT), a novel representation that enables efficient reasoning for imitation learning models. NEAT is a continuous function which maps locations in Bird's Eye View (BEV) scene coordinates to waypoints and semantics. In a new evaluation setting involving adverse environmental conditions and challenging scenarios, NEAT outperforms several strong baselines and achieves driving scores on par with the privileged CARLA expert.
  arXiv  Detail & Related papers  (2021-09-09T17:55:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.