LDM-ISP: Enhancing Neural ISP for Low Light with Latent Diffusion Models

• URL: http://arxiv.org/abs/2312.01027v3
• Date: Wed, 20 Mar 2024 03:19:41 GMT
• Title: LDM-ISP: Enhancing Neural ISP for Low Light with Latent Diffusion Models
• Authors: Qiang Wen, Yazhou Xing, Zhefan Rao, Qifeng Chen,
• Abstract summary: We propose to leverage the pre-trained latent diffusion model to perform the neural ISP for enhancing extremely low-light images. Specifically, to tailor the pre-trained latent diffusion model to operate on the RAW domain, we train a set of lightweight taming modules. We observe different roles of UNet denoising and decoder reconstruction in the latent diffusion model, which inspires us to decompose the low-light image enhancement task into latent-space low-frequency content generation and decoding-phase high-frequency detail maintenance.
• Score: 54.93010869546011
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Enhancing a low-light noisy RAW image into a well-exposed and clean sRGB image is a significant challenge for modern digital cameras. Prior approaches have difficulties in recovering fine-grained details and true colors of the scene under extremely low-light environments due to near-to-zero SNR. Meanwhile, diffusion models have shown significant progress towards general domain image generation. In this paper, we propose to leverage the pre-trained latent diffusion model to perform the neural ISP for enhancing extremely low-light images. Specifically, to tailor the pre-trained latent diffusion model to operate on the RAW domain, we train a set of lightweight taming modules to inject the RAW information into the diffusion denoising process via modulating the intermediate features of UNet. We further observe different roles of UNet denoising and decoder reconstruction in the latent diffusion model, which inspires us to decompose the low-light image enhancement task into latent-space low-frequency content generation and decoding-phase high-frequency detail maintenance. Through extensive experiments on representative datasets, we demonstrate our simple design not only achieves state-of-the-art performance in quantitative evaluations but also shows significant superiority in visual comparisons over strong baselines, which highlight the effectiveness of powerful generative priors for neural ISP under extremely low-light environments. The project page is available at https://csqiangwen.github.io/projects/ldm-isp/

Related papers

• Unsupervised Low-light Image Enhancement with Lookup Tables and Diffusion Priors [38.96909959677438]
  Low-light image enhancement (LIE) aims at precisely and efficiently recovering an image degraded in poor illumination environments. Recent advanced LIE techniques are using deep neural networks, which require lots of low-normal light image pairs, network parameters, and computational resources. We devise a novel unsupervised LIE framework based on diffusion priors and lookup tables to achieve efficient low-light image recovery.
  arXiv  Detail & Related papers  (2024-09-27T16:37:27Z)
• ExposureDiffusion: Learning to Expose for Low-light Image Enhancement [87.08496758469835]
  This work addresses the issue by seamlessly integrating a diffusion model with a physics-based exposure model. Our method obtains significantly improved performance and reduced inference time compared with vanilla diffusion models. The proposed framework can work with both real-paired datasets, SOTA noise models, and different backbone networks.
  arXiv  Detail & Related papers  (2023-07-15T04:48:35Z)
• Low-Light Image Enhancement with Wavelet-based Diffusion Models [50.632343822790006]
  Diffusion models have achieved promising results in image restoration tasks, yet suffer from time-consuming, excessive computational resource consumption, and unstable restoration. We propose a robust and efficient Diffusion-based Low-Light image enhancement approach, dubbed DiffLL.
  arXiv  Detail & Related papers  (2023-06-01T03:08:28Z)
• Instance Segmentation in the Dark [43.85818645776587]
  We take a deep look at instance segmentation in the dark and introduce several techniques that substantially boost the low-light inference accuracy. We propose a novel learning method that relies on an adaptive weighted downsampling layer, a smooth-oriented convolutional block, and disturbance suppression learning. We capture a real-world low-light instance segmentation dataset comprising over two thousand paired low/normal-light images with instance-level pixel-wise annotations.
  arXiv  Detail & Related papers  (2023-04-27T16:02:29Z)
• Denoising Diffusion Post-Processing for Low-Light Image Enhancement [0.0]
  Low-light image enhancement (LLIE) techniques attempt to increase the visibility of images captured in low-light scenarios. LLIE techniques introduce a variety of image degradations such as noise and color bias. Post-processing denoisers have widely been used, which often yield oversmoothed results lacking detail. We introduce Low-light Post-processing Diffusion Model (LPDM) in order to model the conditional distribution between under-exposed and normally-exposed images.
  arXiv  Detail & Related papers  (2023-03-16T20:06:55Z)
• Seeing Through The Noisy Dark: Toward Real-world Low-Light Image Enhancement and Denoising [125.56062454927755]
  Real-world low-light environment usually suffer from lower visibility and heavier noise, due to insufficient light or hardware limitation. We propose a novel end-to-end method termed Real-world Low-light Enhancement & Denoising Network (RLED-Net)
  arXiv  Detail & Related papers  (2022-10-02T14:57:23Z)
• INFWIDE: Image and Feature Space Wiener Deconvolution Network for Non-blind Image Deblurring in Low-Light Conditions [32.35378513394865]
  We propose a novel non-blind deblurring method dubbed image and feature space Wiener deconvolution network (INFWIDE) INFWIDE removes noise and hallucinates saturated regions in the image space and suppresses ringing artifacts in the feature space. Experiments on synthetic data and real data demonstrate the superior performance of the proposed approach.
  arXiv  Detail & Related papers  (2022-07-17T15:22:31Z)
• Cycle-Interactive Generative Adversarial Network for Robust Unsupervised Low-Light Enhancement [109.335317310485]
  Cycle-Interactive Generative Adversarial Network (CIGAN) is capable of not only better transferring illumination distributions between low/normal-light images but also manipulating detailed signals. In particular, the proposed low-light guided transformation feed-forwards the features of low-light images from the generator of enhancement GAN into the generator of degradation GAN.
  arXiv  Detail & Related papers  (2022-07-03T06:37:46Z)
• Unsupervised Low-light Image Enhancement with Decoupled Networks [103.74355338972123]
  We learn a two-stage GAN-based framework to enhance the real-world low-light images in a fully unsupervised fashion. Our proposed method outperforms the state-of-the-art unsupervised image enhancement methods in terms of both illumination enhancement and noise reduction.
  arXiv  Detail & Related papers  (2020-05-06T13:37:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.