FlatNet: Towards Photorealistic Scene Reconstruction from Lensless Measurements

• URL: http://arxiv.org/abs/2010.15440v1
• Date: Thu, 29 Oct 2020 09:20:22 GMT
• Title: FlatNet: Towards Photorealistic Scene Reconstruction from Lensless Measurements
• Authors: Salman S. Khan, Varun Sundar, Vivek Boominathan, Ashok Veeraraghavan, and Kaushik Mitra
• Abstract summary: We propose a non-iterative deep learning based reconstruction approach that results in orders of magnitude improvement in image quality for lensless reconstructions. Our approach, called $textitFlatNet$, lays down a framework for reconstructing high-quality photorealistic images from mask-based lensless cameras.
• Score: 31.353395064815892
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Lensless imaging has emerged as a potential solution towards realizing ultra-miniature cameras by eschewing the bulky lens in a traditional camera. Without a focusing lens, the lensless cameras rely on computational algorithms to recover the scenes from multiplexed measurements. However, the current iterative-optimization-based reconstruction algorithms produce noisier and perceptually poorer images. In this work, we propose a non-iterative deep learning based reconstruction approach that results in orders of magnitude improvement in image quality for lensless reconstructions. Our approach, called $\textit{FlatNet}$, lays down a framework for reconstructing high-quality photorealistic images from mask-based lensless cameras, where the camera's forward model formulation is known. FlatNet consists of two stages: (1) an inversion stage that maps the measurement into a space of intermediate reconstruction by learning parameters within the forward model formulation, and (2) a perceptual enhancement stage that improves the perceptual quality of this intermediate reconstruction. These stages are trained together in an end-to-end manner. We show high-quality reconstructions by performing extensive experiments on real and challenging scenes using two different types of lensless prototypes: one which uses a separable forward model and another, which uses a more general non-separable cropped-convolution model. Our end-to-end approach is fast, produces photorealistic reconstructions, and is easy to adopt for other mask-based lensless cameras.

Related papers

• GANESH: Generalizable NeRF for Lensless Imaging [12.985055542373791]
  We introduce GANESH, a novel framework designed to enable simultaneous refinement and novel view synthesis from lensless images. Unlike existing methods that require scene-specific training, our approach supports on-the-fly inference without retraining on each scene. To facilitate research in this area, we also present the first multi-view lensless dataset, LenslessScenes.
  arXiv  Detail & Related papers  (2024-11-07T15:47:07Z)
• PhoCoLens: Photorealistic and Consistent Reconstruction in Lensless Imaging [19.506766336040247]
  Lensless cameras offer significant advantages in size, weight, and cost compared to traditional lens-based systems. Current algorithms struggle with inaccurate forward imaging models and insufficient priors to reconstruct high-quality images. We introduce a novel two-stage approach for consistent and photorealistic lensless image reconstruction.
  arXiv  Detail & Related papers  (2024-09-26T16:07:24Z)
• DifuzCam: Replacing Camera Lens with a Mask and a Diffusion Model [31.43307762723943]
  The flat lensless camera design reduces the camera size and weight significantly. The image is recovered from the raw sensor measurements using a reconstruction algorithm. We propose utilizing a pre-trained diffusion model with a control network and a learned separable transformation for reconstruction.
  arXiv  Detail & Related papers  (2024-08-14T13:20:52Z)
• Optical Aberration Correction in Postprocessing using Imaging Simulation [17.331939025195478]
  The popularity of mobile photography continues to grow. Recent cameras have shifted some of these correction tasks from optical design to postprocessing systems. We propose a practical method for recovering the degradation caused by optical aberrations.
  arXiv  Detail & Related papers  (2023-05-10T03:20:39Z)
• Neural Lens Modeling [50.57409162437732]
  NeuroLens is a neural lens model for distortion and vignetting that can be used for point projection and ray casting. It can be used to perform pre-capture calibration using classical calibration targets, and can later be used to perform calibration or refinement during 3D reconstruction. The model generalizes across many lens types and is trivial to integrate into existing 3D reconstruction and rendering systems.
  arXiv  Detail & Related papers  (2023-04-10T20:09:17Z)
• Neural 3D Reconstruction in the Wild [86.6264706256377]
  We introduce a new method that enables efficient and accurate surface reconstruction from Internet photo collections. We present a new benchmark and protocol for evaluating reconstruction performance on such in-the-wild scenes.
  arXiv  Detail & Related papers  (2022-05-25T17:59:53Z)
• Unrolled Primal-Dual Networks for Lensless Cameras [0.45880283710344055]
  We show that learning a supervised primal-dual reconstruction method results in image quality matching state of the art in the literature. This improvement stems from our finding that embedding learnable forward and adjoint models in a learned primal-dual optimization framework can even improve the quality of reconstructed images.
  arXiv  Detail & Related papers  (2022-03-08T19:21:39Z)
• Towards Non-Line-of-Sight Photography [48.491977359971855]
  Non-line-of-sight (NLOS) imaging is based on capturing the multi-bounce indirect reflections from the hidden objects. Active NLOS imaging systems rely on the capture of the time of flight of light through the scene. We propose a new problem formulation, called NLOS photography, to specifically address this deficiency.
  arXiv  Detail & Related papers  (2021-09-16T08:07:13Z)
• Universal and Flexible Optical Aberration Correction Using Deep-Prior Based Deconvolution [51.274657266928315]
  We propose a PSF aware plug-and-play deep network, which takes the aberrant image and PSF map as input and produces the latent high quality version via incorporating lens-specific deep priors. Specifically, we pre-train a base model from a set of diverse lenses and then adapt it to a given lens by quickly refining the parameters.
  arXiv  Detail & Related papers  (2021-04-07T12:00:38Z)
• Towards High Fidelity Monocular Face Reconstruction with Rich Reflectance using Self-supervised Learning and Ray Tracing [49.759478460828504]
  Methods combining deep neural network encoders with differentiable rendering have opened up the path for very fast monocular reconstruction of geometry, lighting and reflectance. ray tracing was introduced for monocular face reconstruction within a classic optimization-based framework. We propose a new method that greatly improves reconstruction quality and robustness in general scenes.
  arXiv  Detail & Related papers  (2021-03-29T08:58:10Z)
• Vid2Curve: Simultaneous Camera Motion Estimation and Thin Structure Reconstruction from an RGB Video [90.93141123721713]
  Thin structures, such as wire-frame sculptures, fences, cables, power lines, and tree branches, are common in the real world. It is extremely challenging to acquire their 3D digital models using traditional image-based or depth-based reconstruction methods because thin structures often lack distinct point features and have severe self-occlusion. We propose the first approach that simultaneously estimates camera motion and reconstructs the geometry of complex 3D thin structures in high quality from a color video captured by a handheld camera.
  arXiv  Detail & Related papers  (2020-05-07T10:39:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.