UniDrive: Towards Universal Driving Perception Across Camera Configurations

• URL: http://arxiv.org/abs/2410.13864v1
• Date: Thu, 17 Oct 2024 17:59:59 GMT
• Title: UniDrive: Towards Universal Driving Perception Across Camera Configurations
• Authors: Ye Li, Wenzhao Zheng, Xiaonan Huang, Kurt Keutzer,
• Abstract summary: 3D perception aims to infer 3D information from 2D images based on 3D-2D projection. Generalizing across camera configurations is important for deploying autonomous driving models on different car models. We present UniDrive, a novel framework for vision-centric autonomous driving to achieve universal perception across camera configurations.
• Score: 38.40168936403638
• License:
• Abstract: Vision-centric autonomous driving has demonstrated excellent performance with economical sensors. As the fundamental step, 3D perception aims to infer 3D information from 2D images based on 3D-2D projection. This makes driving perception models susceptible to sensor configuration (e.g., camera intrinsics and extrinsics) variations. However, generalizing across camera configurations is important for deploying autonomous driving models on different car models. In this paper, we present UniDrive, a novel framework for vision-centric autonomous driving to achieve universal perception across camera configurations. We deploy a set of unified virtual cameras and propose a ground-aware projection method to effectively transform the original images into these unified virtual views. We further propose a virtual configuration optimization method by minimizing the expected projection error between original cameras and virtual cameras. The proposed virtual camera projection can be applied to existing 3D perception methods as a plug-and-play module to mitigate the challenges posed by camera parameter variability, resulting in more adaptable and reliable driving perception models. To evaluate the effectiveness of our framework, we collect a dataset on Carla by driving the same routes while only modifying the camera configurations. Experimental results demonstrate that our method trained on one specific camera configuration can generalize to varying configurations with minor performance degradation.

Related papers

• Multi-camera Bird's Eye View Perception for Autonomous Driving [17.834495597639805]
  It is essential to produce perception outputs in 3D to enable the spatial reasoning of other agents and structures. The most basic approach to achieving the desired BEV representation from a camera image is IPM, assuming a flat ground surface. More recent approaches use deep neural networks to output directly in BEV space.
  arXiv  Detail & Related papers  (2023-09-16T19:12:05Z)
• 3D Data Augmentation for Driving Scenes on Camera [50.41413053812315]
  We propose a 3D data augmentation approach termed Drive-3DAug, aiming at augmenting the driving scenes on camera in the 3D space. We first utilize Neural Radiance Field (NeRF) to reconstruct the 3D models of background and foreground objects. Then, augmented driving scenes can be obtained by placing the 3D objects with adapted location and orientation at the pre-defined valid region of backgrounds.
  arXiv  Detail & Related papers  (2023-03-18T05:51:05Z)
• SurroundDepth: Entangling Surrounding Views for Self-Supervised Multi-Camera Depth Estimation [101.55622133406446]
  We propose a SurroundDepth method to incorporate the information from multiple surrounding views to predict depth maps across cameras. Specifically, we employ a joint network to process all the surrounding views and propose a cross-view transformer to effectively fuse the information from multiple views. In experiments, our method achieves the state-of-the-art performance on the challenging multi-camera depth estimation datasets.
  arXiv  Detail & Related papers  (2022-04-07T17:58:47Z)
• Rope3D: TheRoadside Perception Dataset for Autonomous Driving and Monocular 3D Object Detection Task [48.555440807415664]
  We present the first high-diversity challenging Roadside Perception 3D dataset- Rope3D from a novel view. The dataset consists of 50k images and over 1.5M 3D objects in various scenes. We propose to leverage the geometry constraint to solve the inherent ambiguities caused by various sensors, viewpoints.
  arXiv  Detail & Related papers  (2022-03-25T12:13:23Z)
• SVDistNet: Self-Supervised Near-Field Distance Estimation on Surround View Fisheye Cameras [30.480562747903186]
  A 360deg perception of scene geometry is essential for automated driving, notably for parking and urban driving scenarios. We present novel camera-geometry adaptive multi-scale convolutions which utilize the camera parameters as a conditional input. We evaluate our approach on the Fisheye WoodScape surround-view dataset, significantly improving over previous approaches.
  arXiv  Detail & Related papers  (2021-04-09T15:20:20Z)
• Infrastructure-based Multi-Camera Calibration using Radial Projections [117.22654577367246]
  Pattern-based calibration techniques can be used to calibrate the intrinsics of the cameras individually. Infrastucture-based calibration techniques are able to estimate the extrinsics using 3D maps pre-built via SLAM or Structure-from-Motion. We propose to fully calibrate a multi-camera system from scratch using an infrastructure-based approach.
  arXiv  Detail & Related papers  (2020-07-30T09:21:04Z)
• 3D Scene Geometry-Aware Constraint for Camera Localization with Deep Learning [11.599633757222406]
  Recently end-to-end approaches based on convolutional neural network have been much studied to achieve or even exceed 3D-geometry based traditional methods. In this work, we propose a compact network for absolute camera pose regression. Inspired from those traditional methods, a 3D scene geometry-aware constraint is also introduced by exploiting all available information including motion, depth and image contents.
  arXiv  Detail & Related papers  (2020-05-13T04:15:14Z)
• Lightweight Multi-View 3D Pose Estimation through Camera-Disentangled Representation [57.11299763566534]
  We present a solution to recover 3D pose from multi-view images captured with spatially calibrated cameras. We exploit 3D geometry to fuse input images into a unified latent representation of pose, which is disentangled from camera view-points. Our architecture then conditions the learned representation on camera projection operators to produce accurate per-view 2d detections.
  arXiv  Detail & Related papers  (2020-04-05T12:52:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.