Day-to-Night Image Synthesis for Training Nighttime Neural ISPs

• URL: http://arxiv.org/abs/2206.02715v1
• Date: Mon, 6 Jun 2022 16:15:45 GMT
• Title: Day-to-Night Image Synthesis for Training Nighttime Neural ISPs
• Authors: Abhijith Punnappurath, Abdullah Abuolaim, Abdelrahman Abdelhamed, Alex Levinshtein and Michael S. Brown
• Abstract summary: We propose a method that synthesizes nighttime images from daytime images. Daytime images are easy to capture, exhibit low-noise and rarely suffer from motion blur. We show the effectiveness of our synthesis framework by training neural ISPs for nightmode rendering.
• Score: 39.37467397777888
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Many flagship smartphone cameras now use a dedicated neural image signal processor (ISP) to render noisy raw sensor images to the final processed output. Training nightmode ISP networks relies on large-scale datasets of image pairs with: (1) a noisy raw image captured with a short exposure and a high ISO gain; and (2) a ground truth low-noise raw image captured with a long exposure and low ISO that has been rendered through the ISP. Capturing such image pairs is tedious and time-consuming, requiring careful setup to ensure alignment between the image pairs. In addition, ground truth images are often prone to motion blur due to the long exposure. To address this problem, we propose a method that synthesizes nighttime images from daytime images. Daytime images are easy to capture, exhibit low-noise (even on smartphone cameras) and rarely suffer from motion blur. We outline a processing framework to convert daytime raw images to have the appearance of realistic nighttime raw images with different levels of noise. Our procedure allows us to easily produce aligned noisy and clean nighttime image pairs. We show the effectiveness of our synthesis framework by training neural ISPs for nightmode rendering. Furthermore, we demonstrate that using our synthetic nighttime images together with small amounts of real data (e.g., 5% to 10%) yields performance almost on par with training exclusively on real nighttime images. Our dataset and code are available at https://github.com/SamsungLabs/day-to-night.

Related papers

• Exploring Reliable Matching with Phase Enhancement for Night-time Semantic Segmentation [58.180226179087086]
  We propose a novel end-to-end optimized approach, named NightFormer, tailored for night-time semantic segmentation. Specifically, we design a pixel-level texture enhancement module to acquire texture-aware features hierarchically with phase enhancement and amplified attention. Our proposed method performs favorably against state-of-the-art night-time semantic segmentation methods.
  arXiv  Detail & Related papers  (2024-08-25T13:59:31Z)
• Contrastive Learning for Low-light Raw Denoising [2.929093799984392]
  We introduce a new denoising contrastive regularization (DCR) to exploit the information of noisy images and clean images. In the feature space, DCR makes the denoised image closer to the clean image and far away from the noisy image. In addition, we build a new feature embedding network called Wnet, which is more effective to extract high-frequency information.
  arXiv  Detail & Related papers  (2023-05-05T08:13:53Z)
• D2HNet: Joint Denoising and Deblurring with Hierarchical Network for Robust Night Image Restoration [17.6005530147501]
  Night imaging with modern smartphone cameras is troublesome due to low photon count and unavoidable noise in the imaging system. We develop a D2HNet framework to recover a high-quality image by deblurring and enhancing a long-exposure image under the guidance of a short-exposure image. The results on the validation set and real photos demonstrate our methods achieve better visual quality and state-of-the-art quantitative scores.
  arXiv  Detail & Related papers  (2022-07-07T13:42:05Z)
• When the Sun Goes Down: Repairing Photometric Losses for All-Day Depth Estimation [47.617222712429026]
  We show how to use a combination of three techniques to allow the existing photometric losses to work for both day and nighttime images. First, we introduce a per-pixel neural intensity transformation to compensate for the light changes that occur between successive frames. Second, we predict a per-pixel residual flow map that we use to correct the reprojection correspondences induced by the estimated ego-motion and depth.
  arXiv  Detail & Related papers  (2022-06-28T09:29:55Z)
• Rendering Nighttime Image Via Cascaded Color and Brightness Compensation [22.633061635144887]
  We build a high-resolution nighttime RAW-RGB dataset with white balance and tone mapping annotated by experts. We then develop the CBUnet, a two-stage NN ISP to cascade the compensation of color and brightness attributes. Experiments show that our method has better visual quality compared to traditional ISP pipeline.
  arXiv  Detail & Related papers  (2022-04-19T16:15:31Z)
• Pseudo-ISP: Learning Pseudo In-camera Signal Processing Pipeline from A Color Image Denoiser [73.88164217509816]
  We present an unpaired learning scheme to adapt a color image denoiser for handling test images with noise discrepancy. We consider a practical training setting, i.e., a pre-trained denoiser, a set of test noisy images, and an unpaired set of clean images. Pseudo-ISP is effective in synthesizing realistic noisy sRGB images.
  arXiv  Detail & Related papers  (2021-03-18T13:11:28Z)
• Nighttime Dehazing with a Synthetic Benchmark [147.21955799938115]
  We propose a novel synthetic method called 3R to simulate nighttime hazy images from daytime clear images. We generate realistic nighttime hazy images by sampling real-world light colors from a prior empirical distribution. Experiment results demonstrate their superiority over state-of-the-art methods in terms of both image quality and runtime.
  arXiv  Detail & Related papers  (2020-08-10T02:16:46Z)
• Night-time Scene Parsing with a Large Real Dataset [67.11211537439152]
  We aim to address the night-time scene parsing (NTSP) problem, which has two main challenges. To tackle the scarcity of night-time data, we collect a novel labeled dataset, named it NightCity, of 4,297 real night-time images. We also propose an exposure-aware framework to address the NTSP problem through augmenting the segmentation process with explicitly learned exposure features.
  arXiv  Detail & Related papers  (2020-03-15T18:11:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.