Photon-Starved Scene Inference using Single Photon Cameras
- URL: http://arxiv.org/abs/2107.11001v1
- Date: Fri, 23 Jul 2021 02:27:03 GMT
- Title: Photon-Starved Scene Inference using Single Photon Cameras
- Authors: Bhavya Goyal, Mohit Gupta
- Abstract summary: We propose photon scale-space a collection of high-SNR images spanning a wide range of photons-per-pixel (PPP) levels.
We develop training techniques that push images with different illumination levels closer to each other in feature representation space.
Based on the proposed approach, we demonstrate, via simulations and real experiments with a SPAD camera, high-performance on various inference tasks.
- Score: 14.121328731553868
- License: http://creativecommons.org/publicdomain/zero/1.0/
- Abstract: Scene understanding under low-light conditions is a challenging problem. This
is due to the small number of photons captured by the camera and the resulting
low signal-to-noise ratio (SNR). Single-photon cameras (SPCs) are an emerging
sensing modality that are capable of capturing images with high sensitivity.
Despite having minimal read-noise, images captured by SPCs in photon-starved
conditions still suffer from strong shot noise, preventing reliable scene
inference. We propose photon scale-space a collection of high-SNR images
spanning a wide range of photons-per-pixel (PPP) levels (but same scene
content) as guides to train inference model on low photon flux images. We
develop training techniques that push images with different illumination levels
closer to each other in feature representation space. The key idea is that
having a spectrum of different brightness levels during training enables
effective guidance, and increases robustness to shot noise even in extreme
noise cases. Based on the proposed approach, we demonstrate, via simulations
and real experiments with a SPAD camera, high-performance on various inference
tasks such as image classification and monocular depth estimation under ultra
low-light, down to < 1 PPP.
Related papers
- Radiance Fields from Photons [18.15183252935672]
We introduce quanta radiance fields, a class of neural radiance fields that are trained at the granularity of individual photons using single-photon cameras (SPCs)
We demonstrate, both via simulations and a prototype SPC hardware, high-fidelity reconstructions under high-speed motion, in low light, and for extreme dynamic range settings.
arXiv Detail & Related papers (2024-07-12T16:06:51Z) - LDM-ISP: Enhancing Neural ISP for Low Light with Latent Diffusion Models [54.93010869546011]
We propose to leverage the pre-trained latent diffusion model to perform the neural ISP for enhancing extremely low-light images.
Specifically, to tailor the pre-trained latent diffusion model to operate on the RAW domain, we train a set of lightweight taming modules.
We observe different roles of UNet denoising and decoder reconstruction in the latent diffusion model, which inspires us to decompose the low-light image enhancement task into latent-space low-frequency content generation and decoding-phase high-frequency detail maintenance.
arXiv Detail & Related papers (2023-12-02T04:31:51Z) - Enhancing Low-Light Images Using Infrared-Encoded Images [81.8710581927427]
Previous arts mainly focus on the low-light images captured in the visible spectrum using pixel-wise loss.
We propose a novel approach to increase the visibility of images captured under low-light environments by removing the in-camera infrared (IR) cut-off filter.
arXiv Detail & Related papers (2023-07-09T08:29:19Z) - An Effective Image Restorer: Denoising and Luminance Adjustment for
Low-photon-count Imaging [6.358214877782411]
raw image restoration under low-photon-count conditions by simulating the imaging of quanta image sensor (QIS)
We develop a lightweight framework, which consists of a multi-level pyramid denoising network (MPDNet) and a luminance adjustment (LA) module to achieve separate denoising and luminance enhancement.
Our image restorer can achieve superior performance on the degraded images with various photon levels by suppressing noise and recovering luminance and color effectively.
arXiv Detail & Related papers (2021-10-29T12:16:30Z) - Physics-based Noise Modeling for Extreme Low-light Photography [63.65570751728917]
We study the noise statistics in the imaging pipeline of CMOS photosensors.
We formulate a comprehensive noise model that can accurately characterize the real noise structures.
Our noise model can be used to synthesize realistic training data for learning-based low-light denoising algorithms.
arXiv Detail & Related papers (2021-08-04T16:36:29Z) - Dynamic Low-light Imaging with Quanta Image Sensors [79.28256402267034]
We propose a solution using Quanta Image Sensors (QIS) and present a new image reconstruction algorithm.
We show that dynamic scenes can be reconstructed from a burst of frames at a photon level of 1 photon per pixel per frame.
arXiv Detail & Related papers (2020-07-16T20:29:52Z) - Deep Bilateral Retinex for Low-Light Image Enhancement [96.15991198417552]
Low-light images suffer from poor visibility caused by low contrast, color distortion and measurement noise.
This paper proposes a deep learning method for low-light image enhancement with a particular focus on handling the measurement noise.
The proposed method is very competitive to the state-of-the-art methods, and has significant advantage over others when processing images captured in extremely low lighting conditions.
arXiv Detail & Related papers (2020-07-04T06:26:44Z) - Quanta Burst Photography [15.722085082004934]
Single-photon avalanche diodes (SPADs) are an emerging sensor technology capable of detecting individual incident photons.
We present quanta burst photography, a computational photography technique that leverages SPCs as passive imaging devices for photography in challenging conditions.
arXiv Detail & Related papers (2020-06-21T16:20:29Z) - Unsupervised Low-light Image Enhancement with Decoupled Networks [103.74355338972123]
We learn a two-stage GAN-based framework to enhance the real-world low-light images in a fully unsupervised fashion.
Our proposed method outperforms the state-of-the-art unsupervised image enhancement methods in terms of both illumination enhancement and noise reduction.
arXiv Detail & Related papers (2020-05-06T13:37:08Z) - Learning an Adaptive Model for Extreme Low-light Raw Image Processing [5.706764509663774]
We propose an adaptive low-light raw image enhancement network to improve image quality.
The proposed method has the lowest Noise Level Estimation (NLE) score compared with the state-of-the-art low-light algorithms.
The potential application in video processing is briefly discussed.
arXiv Detail & Related papers (2020-04-22T09:01:07Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.