DeProCams: Simultaneous Relighting, Compensation and Shape Reconstruction for Projector-Camera Systems

• URL: http://arxiv.org/abs/2003.03040v2
• Date: Sun, 24 Jan 2021 09:19:14 GMT
• Title: DeProCams: Simultaneous Relighting, Compensation and Shape Reconstruction for Projector-Camera Systems
• Authors: Bingyao Huang and Haibin Ling
• Abstract summary: We propose a novel end-to-end trainable model named DeProCams to learn the photometric and geometric mappings of ProCams. DeProCams explicitly decomposes the projector-camera image mappings into three subprocesses: shading attributes estimation, rough direct light estimation and photorealistic neural rendering. In our experiments, DeProCams shows clear advantages over previous arts with promising quality and being fully differentiable.
• Score: 91.45207885902786
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Image-based relighting, projector compensation and depth/normal reconstruction are three important tasks of projector-camera systems (ProCams) and spatial augmented reality (SAR). Although they share a similar pipeline of finding projector-camera image mappings, in tradition, they are addressed independently, sometimes with different prerequisites, devices and sampling images. In practice, this may be cumbersome for SAR applications to address them one-by-one. In this paper, we propose a novel end-to-end trainable model named DeProCams to explicitly learn the photometric and geometric mappings of ProCams, and once trained, DeProCams can be applied simultaneously to the three tasks. DeProCams explicitly decomposes the projector-camera image mappings into three subprocesses: shading attributes estimation, rough direct light estimation and photorealistic neural rendering. A particular challenge addressed by DeProCams is occlusion, for which we exploit epipolar constraint and propose a novel differentiable projector direct light mask. Thus, it can be learned end-to-end along with the other modules. Afterwards, to improve convergence, we apply photometric and geometric constraints such that the intermediate results are plausible. In our experiments, DeProCams shows clear advantages over previous arts with promising quality and meanwhile being fully differentiable. Moreover, by solving the three tasks in a unified model, DeProCams waives the need for additional optical devices, radiometric calibrations and structured light.

Related papers

• Lite2Relight: 3D-aware Single Image Portrait Relighting [87.62069509622226]
  Lite2Relight is a novel technique that can predict 3D consistent head poses of portraits. By utilizing a pre-trained geometry-aware encoder and a feature alignment module, we map input images into a relightable 3D space. This includes producing 3D-consistent results of the full head, including hair, eyes, and expressions.
  arXiv  Detail & Related papers  (2024-07-15T07:16:11Z)
• Neural Radiance Fields with Torch Units [19.927273454898295]
  Learning-based 3D reconstruction methods are widely used in industrial applications. In this paper, we propose a novel inference pattern that encourages single camera ray possessing more contextual information. To summarize, as a torchlight, a ray in our proposed method rendering a patch of image. Thus, we call the proposed method, Torch-NeRF.
  arXiv  Detail & Related papers  (2024-04-03T10:08:55Z)
• Dense Voxel 3D Reconstruction Using a Monocular Event Camera [5.599072208069752]
  Event cameras offer many advantages over conventional frame-based cameras. Their application in 3D reconstruction for VR applications is underexplored. We propose a novel approach for solving dense 3D reconstruction using only a single event camera.
  arXiv  Detail & Related papers  (2023-09-01T10:46:57Z)
• Neural Projection Mapping Using Reflectance Fields [11.74757574153076]
  We introduce a projector into a neural reflectance field that allows to calibrate the projector and photo realistic light editing. Our neural field consists of three neural networks, estimating geometry, material, and transmittance. We believe that neural projection mapping opens up the door to novel and exciting downstream tasks, through the joint optimization of the scene and projection images.
  arXiv  Detail & Related papers  (2023-06-11T05:33:10Z)
• A Simple Baseline for Multi-Camera 3D Object Detection [94.63944826540491]
  3D object detection with surrounding cameras has been a promising direction for autonomous driving. We present SimMOD, a Simple baseline for Multi-camera Object Detection. We conduct extensive experiments on the 3D object detection benchmark of nuScenes to demonstrate the effectiveness of SimMOD.
  arXiv  Detail & Related papers  (2022-08-22T03:38:01Z)
• Camera Calibration through Camera Projection Loss [4.36572039512405]
  We propose a novel method to predict intrinsic (focal length and principal point offset) parameters using an image pair. Unlike existing methods, we proposed a new representation that incorporates camera model equations as a neural network in multi-task learning framework. Our proposed approach achieves better performance with respect to both deep learning-based and traditional methods on 7 out of 10 parameters evaluated.
  arXiv  Detail & Related papers  (2021-10-07T14:03:10Z)
• A Projector-Camera System Using Hybrid Pixels with Projection and Capturing Capabilities [23.217683529089005]
  We propose a novel projector-camera system (ProCams) in which each pixel has both projection and capturing capabilities. Our proposed ProCams solves the difficulty of obtaining precise pixel correspondence between the projector and the camera.
  arXiv  Detail & Related papers  (2021-07-11T13:27:25Z)
• Infrastructure-based Multi-Camera Calibration using Radial Projections [117.22654577367246]
  Pattern-based calibration techniques can be used to calibrate the intrinsics of the cameras individually. Infrastucture-based calibration techniques are able to estimate the extrinsics using 3D maps pre-built via SLAM or Structure-from-Motion. We propose to fully calibrate a multi-camera system from scratch using an infrastructure-based approach.
  arXiv  Detail & Related papers  (2020-07-30T09:21:04Z)
• Lightweight Multi-View 3D Pose Estimation through Camera-Disentangled Representation [57.11299763566534]
  We present a solution to recover 3D pose from multi-view images captured with spatially calibrated cameras. We exploit 3D geometry to fuse input images into a unified latent representation of pose, which is disentangled from camera view-points. Our architecture then conditions the learned representation on camera projection operators to produce accurate per-view 2d detections.
  arXiv  Detail & Related papers  (2020-04-05T12:52:29Z)
• Redesigning SLAM for Arbitrary Multi-Camera Systems [51.81798192085111]
  Adding more cameras to SLAM systems improves robustness and accuracy but complicates the design of the visual front-end significantly. In this work, we aim at an adaptive SLAM system that works for arbitrary multi-camera setups. We adapt a state-of-the-art visual-inertial odometry with these modifications, and experimental results show that the modified pipeline can adapt to a wide range of camera setups.
  arXiv  Detail & Related papers  (2020-03-04T11:44:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.