GANESH: Generalizable NeRF for Lensless Imaging

• URL: http://arxiv.org/abs/2411.04810v1
• Date: Thu, 07 Nov 2024 15:47:07 GMT
• Title: GANESH: Generalizable NeRF for Lensless Imaging
• Authors: Rakesh Raj Madavan, Akshat Kaimal, Badhrinarayanan K V, Vinayak Gupta, Rohit Choudhary, Chandrakala Shanmuganathan, Kaushik Mitra,
• Abstract summary: We introduce GANESH, a novel framework designed to enable simultaneous refinement and novel view synthesis from lensless images. Unlike existing methods that require scene-specific training, our approach supports on-the-fly inference without retraining on each scene. To facilitate research in this area, we also present the first multi-view lensless dataset, LenslessScenes.
• Score: 12.985055542373791
• License:
• Abstract: Lensless imaging offers a significant opportunity to develop ultra-compact cameras by removing the conventional bulky lens system. However, without a focusing element, the sensor's output is no longer a direct image but a complex multiplexed scene representation. Traditional methods have attempted to address this challenge by employing learnable inversions and refinement models, but these methods are primarily designed for 2D reconstruction and do not generalize well to 3D reconstruction. We introduce GANESH, a novel framework designed to enable simultaneous refinement and novel view synthesis from multi-view lensless images. Unlike existing methods that require scene-specific training, our approach supports on-the-fly inference without retraining on each scene. Moreover, our framework allows us to tune our model to specific scenes, enhancing the rendering and refinement quality. To facilitate research in this area, we also present the first multi-view lensless dataset, LenslessScenes. Extensive experiments demonstrate that our method outperforms current approaches in reconstruction accuracy and refinement quality. Code and video results are available at https://rakesh-123-cryp.github.io/Rakesh.github.io/

Related papers

• PhoCoLens: Photorealistic and Consistent Reconstruction in Lensless Imaging [19.506766336040247]
  Lensless cameras offer significant advantages in size, weight, and cost compared to traditional lens-based systems. Current algorithms struggle with inaccurate forward imaging models and insufficient priors to reconstruct high-quality images. We introduce a novel two-stage approach for consistent and photorealistic lensless image reconstruction.
  arXiv  Detail & Related papers  (2024-09-26T16:07:24Z)
• DistillNeRF: Perceiving 3D Scenes from Single-Glance Images by Distilling Neural Fields and Foundation Model Features [65.8738034806085]
  DistillNeRF is a self-supervised learning framework for understanding 3D environments in autonomous driving scenes. Our method is a generalizable feedforward model that predicts a rich neural scene representation from sparse, single-frame multi-view camera inputs.
  arXiv  Detail & Related papers  (2024-06-17T21:15:13Z)
• ReconFusion: 3D Reconstruction with Diffusion Priors [104.73604630145847]
  We present ReconFusion to reconstruct real-world scenes using only a few photos. Our approach leverages a diffusion prior for novel view synthesis, trained on synthetic and multiview datasets. Our method synthesizes realistic geometry and texture in underconstrained regions while preserving the appearance of observed regions.
  arXiv  Detail & Related papers  (2023-12-05T18:59:58Z)
• Reconstructing Continuous Light Field From Single Coded Image [7.937367109582907]
  We propose a method for reconstructing a continuous light field of a target scene from a single observed image. Joint aperture-exposure coding implemented in a camera enables effective embedding of 3-D scene information into an observed image. NeRF-based neural rendering enables high quality view synthesis of a 3-D scene from continuous viewpoints.
  arXiv  Detail & Related papers  (2023-11-16T07:59:01Z)
• Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstruction [77.69363640021503]
  3D-aware image synthesis encompasses a variety of tasks, such as scene generation and novel view synthesis from images. We present SSDNeRF, a unified approach that employs an expressive diffusion model to learn a generalizable prior of neural radiance fields (NeRF) from multi-view images of diverse objects.
  arXiv  Detail & Related papers  (2023-04-13T17:59:01Z)
• Neural Lens Modeling [50.57409162437732]
  NeuroLens is a neural lens model for distortion and vignetting that can be used for point projection and ray casting. It can be used to perform pre-capture calibration using classical calibration targets, and can later be used to perform calibration or refinement during 3D reconstruction. The model generalizes across many lens types and is trivial to integrate into existing 3D reconstruction and rendering systems.
  arXiv  Detail & Related papers  (2023-04-10T20:09:17Z)
• Neural 3D Reconstruction in the Wild [86.6264706256377]
  We introduce a new method that enables efficient and accurate surface reconstruction from Internet photo collections. We present a new benchmark and protocol for evaluating reconstruction performance on such in-the-wild scenes.
  arXiv  Detail & Related papers  (2022-05-25T17:59:53Z)
• Unrolled Primal-Dual Networks for Lensless Cameras [0.45880283710344055]
  We show that learning a supervised primal-dual reconstruction method results in image quality matching state of the art in the literature. This improvement stems from our finding that embedding learnable forward and adjoint models in a learned primal-dual optimization framework can even improve the quality of reconstructed images.
  arXiv  Detail & Related papers  (2022-03-08T19:21:39Z)
• Towards Non-Line-of-Sight Photography [48.491977359971855]
  Non-line-of-sight (NLOS) imaging is based on capturing the multi-bounce indirect reflections from the hidden objects. Active NLOS imaging systems rely on the capture of the time of flight of light through the scene. We propose a new problem formulation, called NLOS photography, to specifically address this deficiency.
  arXiv  Detail & Related papers  (2021-09-16T08:07:13Z)
• FlatNet: Towards Photorealistic Scene Reconstruction from Lensless Measurements [31.353395064815892]
  We propose a non-iterative deep learning based reconstruction approach that results in orders of magnitude improvement in image quality for lensless reconstructions. Our approach, called $textitFlatNet$, lays down a framework for reconstructing high-quality photorealistic images from mask-based lensless cameras.
  arXiv  Detail & Related papers  (2020-10-29T09:20:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.