STEPS: Joint Self-supervised Nighttime Image Enhancement and Depth Estimation

• URL: http://arxiv.org/abs/2302.01334v1
• Date: Thu, 2 Feb 2023 18:59:47 GMT
• Title: STEPS: Joint Self-supervised Nighttime Image Enhancement and Depth Estimation
• Authors: Yupeng Zheng, Chengliang Zhong, Pengfei Li, Huan-ang Gao, Yuhang Zheng, Bu Jin, Ling Wang, Hao Zhao, Guyue Zhou, Qichao Zhang and Dongbin Zhao
• Abstract summary: We propose a method that jointly learns a nighttime image enhancer and a depth estimator, without using ground truth for either task. Our method tightly entangles two self-supervised tasks using a newly proposed uncertain pixel masking strategy. We benchmark the method on two established datasets: nuScenes and RobotCar.
• Score: 12.392842482031558
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Self-supervised depth estimation draws a lot of attention recently as it can promote the 3D sensing capabilities of self-driving vehicles. However, it intrinsically relies upon the photometric consistency assumption, which hardly holds during nighttime. Although various supervised nighttime image enhancement methods have been proposed, their generalization performance in challenging driving scenarios is not satisfactory. To this end, we propose the first method that jointly learns a nighttime image enhancer and a depth estimator, without using ground truth for either task. Our method tightly entangles two self-supervised tasks using a newly proposed uncertain pixel masking strategy. This strategy originates from the observation that nighttime images not only suffer from underexposed regions but also from overexposed regions. By fitting a bridge-shaped curve to the illumination map distribution, both regions are suppressed and two tasks are bridged naturally. We benchmark the method on two established datasets: nuScenes and RobotCar and demonstrate state-of-the-art performance on both of them. Detailed ablations also reveal the mechanism of our proposal. Last but not least, to mitigate the problem of sparse ground truth of existing datasets, we provide a new photo-realistically enhanced nighttime dataset based upon CARLA. It brings meaningful new challenges to the community. Codes, data, and models are available at https://github.com/ucaszyp/STEPS.

Related papers

• Exploring Reliable Matching with Phase Enhancement for Night-time Semantic Segmentation [58.180226179087086]
  We propose a novel end-to-end optimized approach, named NightFormer, tailored for night-time semantic segmentation. Specifically, we design a pixel-level texture enhancement module to acquire texture-aware features hierarchically with phase enhancement and amplified attention. Our proposed method performs favorably against state-of-the-art night-time semantic segmentation methods.
  arXiv  Detail & Related papers  (2024-08-25T13:59:31Z)
• Sun Off, Lights On: Photorealistic Monocular Nighttime Simulation for Robust Semantic Perception [53.631644875171595]
  Nighttime scenes are hard to semantically perceive with learned models and annotate for humans. Our method, named Sun Off, Lights On (SOLO), is the first to perform nighttime simulation on single images in a photorealistic fashion by operating in 3D. Not only is the visual quality and photorealism of our nighttime images superior to competing approaches including diffusion models, but the former images are also proven more beneficial for semantic nighttime segmentation in day-to-night adaptation.
  arXiv  Detail & Related papers  (2024-07-29T18:00:09Z)
• Self-Supervised Monocular Depth Estimation in the Dark: Towards Data Distribution Compensation [24.382795861986803]
  Using night images for self-supervision is unreliable because the photometric consistency assumption is usually violated in the videos taken under complex lighting conditions. We propose a self-supervised nighttime monocular depth estimation method that does not use any night images during training.
  arXiv  Detail & Related papers  (2024-04-22T03:39:03Z)
• Improving Panoptic Segmentation for Nighttime or Low-Illumination Urban Driving Scenes [0.0]
  We propose two new methods to improve the performance and robustness of Panoptic segmentation. One of the main factors for poor results is the lack of sufficient and accurately annotated nighttime images for urban driving scenes. The proposed approach makes use of CycleGAN to translate daytime images with existing panoptic annotations into nighttime images. It is then utilized to retrain a Panoptic segmentation model to improve performance and robustness under poor illumination and nighttime conditions.
  arXiv  Detail & Related papers  (2023-06-23T18:14:26Z)
• When the Sun Goes Down: Repairing Photometric Losses for All-Day Depth Estimation [47.617222712429026]
  We show how to use a combination of three techniques to allow the existing photometric losses to work for both day and nighttime images. First, we introduce a per-pixel neural intensity transformation to compensate for the light changes that occur between successive frames. Second, we predict a per-pixel residual flow map that we use to correct the reprojection correspondences induced by the estimated ego-motion and depth.
  arXiv  Detail & Related papers  (2022-06-28T09:29:55Z)
• Benchmarking the Robustness of LiDAR-Camera Fusion for 3D Object Detection [58.81316192862618]
  Two critical sensors for 3D perception in autonomous driving are the camera and the LiDAR. fusing these two modalities can significantly boost the performance of 3D perception models. We benchmark the state-of-the-art fusion methods for the first time.
  arXiv  Detail & Related papers  (2022-05-30T09:35:37Z)
• Self-supervised Monocular Depth Estimation for All Day Images using Domain Separation [17.066753214406525]
  We propose a domain-separated network for self-supervised depth estimation of all-day images. Our approach achieves state-of-the-art depth estimation results for all-day images on the challenging Oxford RobotCar dataset.
  arXiv  Detail & Related papers  (2021-08-17T13:52:19Z)
• Regularizing Nighttime Weirdness: Efficient Self-supervised Monocular Depth Estimation in the Dark [20.66405067066299]
  We introduce Priors-Based Regularization to learn distribution knowledge from unpaired depth maps. We also leverage Mapping-Consistent Image Enhancement module to enhance image visibility and contrast. Our framework achieves remarkable improvements and state-of-the-art results on two nighttime datasets.
  arXiv  Detail & Related papers  (2021-08-09T06:24:35Z)
• Unsupervised Low-light Image Enhancement with Decoupled Networks [103.74355338972123]
  We learn a two-stage GAN-based framework to enhance the real-world low-light images in a fully unsupervised fashion. Our proposed method outperforms the state-of-the-art unsupervised image enhancement methods in terms of both illumination enhancement and noise reduction.
  arXiv  Detail & Related papers  (2020-05-06T13:37:08Z)
• Night-time Scene Parsing with a Large Real Dataset [67.11211537439152]
  We aim to address the night-time scene parsing (NTSP) problem, which has two main challenges. To tackle the scarcity of night-time data, we collect a novel labeled dataset, named it NightCity, of 4,297 real night-time images. We also propose an exposure-aware framework to address the NTSP problem through augmenting the segmentation process with explicitly learned exposure features.
  arXiv  Detail & Related papers  (2020-03-15T18:11:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.