DaF-BEVSeg: Distortion-aware Fisheye Camera based Bird's Eye View Segmentation with Occlusion Reasoning

• URL: http://arxiv.org/abs/2404.06352v1
• Date: Tue, 9 Apr 2024 14:43:19 GMT
• Title: DaF-BEVSeg: Distortion-aware Fisheye Camera based Bird's Eye View Segmentation with Occlusion Reasoning
• Authors: Senthil Yogamani, David Unger, Venkatraman Narayanan, Varun Ravi Kumar,
• Abstract summary: There is limited work on BEV segmentation for surround-view fisheye cameras, commonly used in commercial vehicles. We create a synthetic dataset using the Cognata simulator comprising diverse road types, weather, and lighting conditions. We generalize the BEV segmentation to work with any camera model; this is useful for mixing diverse cameras.
• Score: 7.012508171229966
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Semantic segmentation is an effective way to perform scene understanding. Recently, segmentation in 3D Bird's Eye View (BEV) space has become popular as its directly used by drive policy. However, there is limited work on BEV segmentation for surround-view fisheye cameras, commonly used in commercial vehicles. As this task has no real-world public dataset and existing synthetic datasets do not handle amodal regions due to occlusion, we create a synthetic dataset using the Cognata simulator comprising diverse road types, weather, and lighting conditions. We generalize the BEV segmentation to work with any camera model; this is useful for mixing diverse cameras. We implement a baseline by applying cylindrical rectification on the fisheye images and using a standard LSS-based BEV segmentation model. We demonstrate that we can achieve better performance without undistortion, which has the adverse effects of increased runtime due to pre-processing, reduced field-of-view, and resampling artifacts. Further, we introduce a distortion-aware learnable BEV pooling strategy that is more effective for the fisheye cameras. We extend the model with an occlusion reasoning module, which is critical for estimating in BEV space. Qualitative performance of DaF-BEVSeg is showcased in the video at https://streamable.com/ge4v51.

Related papers

• GS-Blur: A 3D Scene-Based Dataset for Realistic Image Deblurring [50.72230109855628]
  We propose GS-Blur, a dataset of synthesized realistic blurry images created using a novel approach. We first reconstruct 3D scenes from multi-view images using 3D Gaussian Splatting (3DGS), then render blurry images by moving the camera view along the randomly generated motion trajectories. By adopting various camera trajectories in reconstructing our GS-Blur, our dataset contains realistic and diverse types of blur, offering a large-scale dataset that generalizes well to real-world blur.
  arXiv  Detail & Related papers  (2024-10-31T06:17:16Z)
• EF-3DGS: Event-Aided Free-Trajectory 3D Gaussian Splatting [76.02450110026747]
  Event cameras, inspired by biological vision, record pixel-wise intensity changes asynchronously with high temporal resolution. We propose Event-Aided Free-Trajectory 3DGS, which seamlessly integrates the advantages of event cameras into 3DGS. We evaluate our method on the public Tanks and Temples benchmark and a newly collected real-world dataset, RealEv-DAVIS.
  arXiv  Detail & Related papers  (2024-10-20T13:44:24Z)
• Redundancy-Aware Camera Selection for Indoor Scene Neural Rendering [54.468355408388675]
  We build a similarity matrix that incorporates both the spatial diversity of the cameras and the semantic variation of the images. We apply a diversity-based sampling algorithm to optimize the camera selection. We also develop a new dataset, IndoorTraj, which includes long and complex camera movements captured by humans in virtual indoor environments.
  arXiv  Detail & Related papers  (2024-09-11T08:36:49Z)
• Towards Viewpoint Robustness in Bird's Eye View Segmentation [85.99907496019972]
  We study how AV perception models are affected by changes in camera viewpoint. Small changes to pitch, yaw, depth, or height of the camera at inference time lead to large drops in performance. We introduce a technique for novel view synthesis and use it to transform collected data to the viewpoint of target rigs.
  arXiv  Detail & Related papers  (2023-09-11T02:10:07Z)
• Semi-Supervised Learning for Visual Bird's Eye View Semantic Segmentation [16.3996408206659]
  We present a novel semi-supervised framework for visual BEV semantic segmentation to boost performance by exploiting unlabeled images during the training. A consistency loss that makes full use of unlabeled data is then proposed to constrain the model on not only semantic prediction but also the BEV feature. Experiments on the nuScenes and Argoverse datasets show that our framework can effectively improve prediction accuracy.
  arXiv  Detail & Related papers  (2023-08-28T12:23:36Z)
• F2BEV: Bird's Eye View Generation from Surround-View Fisheye Camera Images for Automated Driving [3.286961611175469]
  We introduce a baseline, F2BEV, to generate BEV height maps and BEV semantic segmentation maps from fisheye images. F2BEV consists of a distortion-aware spatial cross attention module for querying and consolidating spatial information. We evaluate single-task and multi-task variants of F2BEV on our synthetic FB-SSEM dataset.
  arXiv  Detail & Related papers  (2023-03-07T04:58:57Z)
• Monocular BEV Perception of Road Scenes via Front-to-Top View Projection [57.19891435386843]
  We present a novel framework that reconstructs a local map formed by road layout and vehicle occupancy in the bird's-eye view. Our model runs at 25 FPS on a single GPU, which is efficient and applicable for real-time panorama HD map reconstruction.
  arXiv  Detail & Related papers  (2022-11-15T13:52:41Z)
• BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation [105.96557764248846]
  We introduce BEVFusion, a generic multi-task multi-sensor fusion framework. It unifies multi-modal features in the shared bird's-eye view representation space. It achieves 1.3% higher mAP and NDS on 3D object detection and 13.6% higher mIoU on BEV map segmentation, with 1.9x lower cost.
  arXiv  Detail & Related papers  (2022-05-26T17:59:35Z)
• BEVSegFormer: Bird's Eye View Semantic Segmentation From Arbitrary Camera Rigs [3.5728676902207988]
  We present an effective transformer-based method for BEV semantic segmentation from arbitrary camera rigs. Specifically, our method first encodes image features from arbitrary cameras with a shared backbone. An efficient multi-camera deformable attention unit is designed to carry out the BEV-to-image view transformation.
  arXiv  Detail & Related papers  (2022-03-08T12:39:51Z)
• BEV-MODNet: Monocular Camera based Bird's Eye View Moving Object Detection for Autonomous Driving [2.9769485817170387]
  CNNs can leverage the global context in the scene to project better. We create an extended KITTI-raw dataset consisting of 12.9k images with annotations of moving object masks in BEV space for five classes. We observe a significant improvement of 13% in mIoU using the simple baseline implementation.
  arXiv  Detail & Related papers  (2021-07-11T01:11:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.