GenRadar: Self-supervised Probabilistic Camera Synthesis based on Radar
Frequencies
- URL: http://arxiv.org/abs/2107.08948v1
- Date: Mon, 19 Jul 2021 15:00:28 GMT
- Title: GenRadar: Self-supervised Probabilistic Camera Synthesis based on Radar
Frequencies
- Authors: Carsten Ditzel and Klaus Dietmayer
- Abstract summary: This work combines the complementary strengths of both sensor types in a unique self-learning fusion approach for a probabilistic scene reconstruction.
A proposed algorithm exploits similarities and establishes correspondences between both domains at different feature levels during training.
These discrete tokens are finally transformed back into an instructive view of the respective surrounding, allowing to visually perceive potential dangers.
- Score: 12.707035083920227
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Autonomous systems require a continuous and dependable environment perception
for navigation and decision-making, which is best achieved by combining
different sensor types. Radar continues to function robustly in compromised
circumstances in which cameras become impaired, guaranteeing a steady inflow of
information. Yet, camera images provide a more intuitive and readily applicable
impression of the world. This work combines the complementary strengths of both
sensor types in a unique self-learning fusion approach for a probabilistic
scene reconstruction in adverse surrounding conditions. After reducing the
memory requirements of both high-dimensional measurements through a decoupled
stochastic self-supervised compression technique, the proposed algorithm
exploits similarities and establishes correspondences between both domains at
different feature levels during training. Then, at inference time, relying
exclusively on radio frequencies, the model successively predicts camera
constituents in an autoregressive and self-contained process. These discrete
tokens are finally transformed back into an instructive view of the respective
surrounding, allowing to visually perceive potential dangers for important
tasks downstream.
Related papers
- Fisheye Camera and Ultrasonic Sensor Fusion For Near-Field Obstacle
Perception in Bird's-Eye-View [4.536942273206611]
We present the first end-to-end multimodal fusion model tailored for efficient obstacle perception in a bird's-eye-view (BEV) perspective.
Fisheye cameras are frequently employed for comprehensive surround-view perception, including rear-view obstacle localization.
However, the performance of such cameras can significantly deteriorate in low-light conditions, during nighttime, or when subjected to intense sun glare.
arXiv Detail & Related papers (2024-02-01T14:52:16Z) - Exploring Contextual Representation and Multi-Modality for End-to-End
Autonomous Driving [58.879758550901364]
Recent perception systems enhance spatial understanding with sensor fusion but often lack full environmental context.
We introduce a framework that integrates three cameras to emulate the human field of view, coupled with top-down bird-eye-view semantic data to enhance contextual representation.
Our method achieves displacement error by 0.67m in open-loop settings, surpassing current methods by 6.9% on the nuScenes dataset.
arXiv Detail & Related papers (2022-10-13T05:56:20Z) - Unsupervised Foggy Scene Understanding via Self Spatial-Temporal Label
Diffusion [51.11295961195151]
We exploit the characteristics of the foggy image sequence of driving scenes to densify the confident pseudo labels.
Based on the two discoveries of local spatial similarity and adjacent temporal correspondence of the sequential image data, we propose a novel Target-Domain driven pseudo label Diffusion scheme.
Our scheme helps the adaptive model achieve 51.92% and 53.84% mean intersection-over-union (mIoU) on two publicly available natural foggy datasets.
arXiv Detail & Related papers (2022-06-10T05:16:50Z) - ReDFeat: Recoupling Detection and Description for Multimodal Feature
Learning [51.07496081296863]
We recouple independent constraints of detection and description of multimodal feature learning with a mutual weighting strategy.
We propose a detector that possesses a large receptive field and is equipped with learnable non-maximum suppression layers.
We build a benchmark that contains cross visible, infrared, near-infrared and synthetic aperture radar image pairs for evaluating the performance of features in feature matching and image registration tasks.
arXiv Detail & Related papers (2022-05-16T04:24:22Z) - SurroundDepth: Entangling Surrounding Views for Self-Supervised
Multi-Camera Depth Estimation [101.55622133406446]
We propose a SurroundDepth method to incorporate the information from multiple surrounding views to predict depth maps across cameras.
Specifically, we employ a joint network to process all the surrounding views and propose a cross-view transformer to effectively fuse the information from multiple views.
In experiments, our method achieves the state-of-the-art performance on the challenging multi-camera depth estimation datasets.
arXiv Detail & Related papers (2022-04-07T17:58:47Z) - SSTN: Self-Supervised Domain Adaptation Thermal Object Detection for
Autonomous Driving [6.810856082577402]
We have proposed a deep neural network Self Supervised Thermal Network (SSTN) for learning the feature embedding to maximize the information between visible and infrared spectrum domain by contrastive learning.
The proposed method is extensively evaluated on the two publicly available datasets: the FLIR-ADAS dataset and the KAIST Multi-Spectral dataset.
arXiv Detail & Related papers (2021-03-04T16:42:49Z) - Camera Invariant Feature Learning for Generalized Face Anti-spoofing [95.30490139294136]
We describe a framework that eliminates the influence of inherent variance from acquisition cameras at the feature level.
Experiments show that the proposed method can achieve better performance in both intra-dataset and cross-dataset settings.
arXiv Detail & Related papers (2021-01-25T13:40:43Z) - YOdar: Uncertainty-based Sensor Fusion for Vehicle Detection with Camera
and Radar Sensors [4.396860522241306]
We present an uncertainty-based method for sensor fusion with camera and radar data.
In our experiments we combine the YOLOv3 object detection network with a customized $1D$ radar segmentation network.
Our experiments show, that this approach of uncertainty aware fusion significantly gains performance compared to single sensor baselines.
arXiv Detail & Related papers (2020-10-07T10:40:02Z) - Camera-Lidar Integration: Probabilistic sensor fusion for semantic
mapping [8.18198392834469]
An automated vehicle must be able to perceive and recognise object/obstacles in a three-dimensional world while navigating in a constantly changing environment.
We present a probabilistic pipeline that incorporates uncertainties from the sensor readings (cameras, lidar, IMU and wheel encoders), compensation for the motion of the vehicle, and label probabilities for the semantic images.
arXiv Detail & Related papers (2020-07-09T07:59:39Z) - Semantic sensor fusion: from camera to sparse lidar information [7.489722641968593]
This paper presents an approach to fuse different sensory information, Light Detection and Ranging (lidar) scans and camera images.
The transference of semantic information between the labelled image and the lidar point cloud is performed in four steps.
arXiv Detail & Related papers (2020-03-04T03:09:33Z) - Learning Selective Sensor Fusion for States Estimation [47.76590539558037]
We propose SelectFusion, an end-to-end selective sensor fusion module.
During prediction, the network is able to assess the reliability of the latent features from different sensor modalities.
We extensively evaluate all fusion strategies in both public datasets and on progressively degraded datasets.
arXiv Detail & Related papers (2019-12-30T20:25:16Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.