Learning to Factorize and Relight a City

• URL: http://arxiv.org/abs/2008.02796v1
• Date: Thu, 6 Aug 2020 17:59:54 GMT
• Title: Learning to Factorize and Relight a City
• Authors: Andrew Liu, Shiry Ginosar, Tinghui Zhou, Alexei A. Efros, Noah Snavely
• Abstract summary: We propose a learning-based framework for disentangling outdoor scenes into temporally-varying illumination and permanent scene factors. We show that our learned disentangled factors can be used to manipulate novel images in realistic ways, such as changing lighting effects and scene geometry.
• Score: 70.81496092672421
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose a learning-based framework for disentangling outdoor scenes into temporally-varying illumination and permanent scene factors. Inspired by the classic intrinsic image decomposition, our learning signal builds upon two insights: 1) combining the disentangled factors should reconstruct the original image, and 2) the permanent factors should stay constant across multiple temporal samples of the same scene. To facilitate training, we assemble a city-scale dataset of outdoor timelapse imagery from Google Street View, where the same locations are captured repeatedly through time. This data represents an unprecedented scale of spatio-temporal outdoor imagery. We show that our learned disentangled factors can be used to manipulate novel images in realistic ways, such as changing lighting effects and scene geometry. Please visit factorize-a-city.github.io for animated results.

Related papers

• Neural Scene Chronology [79.51094408119148]
  We aim to reconstruct a time-varying 3D model, capable of rendering photo-realistic renderings with independent control of viewpoint, illumination, and time. In this work, we represent the scene as a space-time radiance field with a per-image illumination embedding, where temporally-varying scene changes are encoded using a set of learned step functions.
  arXiv  Detail & Related papers  (2023-06-13T17:59:58Z)
• Self-Supervised Feature Learning for Long-Term Metric Visual Localization [16.987148593917905]
  We present a novel self-supervised feature learning framework for metric visual localization. We use a sequence-based image matching algorithm to generate image correspondences without ground-truth labels. We can then sample image pairs to train a deep neural network that learns sparse features with associated descriptors and scores without ground-truth pose supervision.
  arXiv  Detail & Related papers  (2022-11-30T21:15:05Z)
• Neural Radiance Transfer Fields for Relightable Novel-view Synthesis with Global Illumination [63.992213016011235]
  We propose a method for scene relighting under novel views by learning a neural precomputed radiance transfer function. Our method can be solely supervised on a set of real images of the scene under a single unknown lighting condition. Results show that the recovered disentanglement of scene parameters improves significantly over the current state of the art.
  arXiv  Detail & Related papers  (2022-07-27T16:07:48Z)
• Factorized and Controllable Neural Re-Rendering of Outdoor Scene for Photo Extrapolation [50.00344639039158]
  We propose a factorized neural re-rendering model to produce novel views from cluttered outdoor Internet photo collections. We also propose a novel realism augmentation process to complement appearance details, which automatically propagates the texture details from a narrow captured photo to the extrapolated neural rendered image.
  arXiv  Detail & Related papers  (2022-07-14T13:28:08Z)
• Multi-view Gradient Consistency for SVBRDF Estimation of Complex Scenes under Natural Illumination [6.282068591820945]
  This paper presents a process for estimating the spatially varying surface reflectance of complex scenes observed under natural illumination. An end-to-end process uses a model of the scene's geometry and several images capturing the scene's surfaces. Experiments show that our technique produces realistic results for arbitrary outdoor scenes with complex geometry.
  arXiv  Detail & Related papers  (2022-02-25T23:49:39Z)
• IBRNet: Learning Multi-View Image-Based Rendering [67.15887251196894]
  We present a method that synthesizes novel views of complex scenes by interpolating a sparse set of nearby views. By drawing on source views at render time, our method hearkens back to classic work on image-based rendering.
  arXiv  Detail & Related papers  (2021-02-25T18:56:21Z)
• Crowdsampling the Plenoptic Function [56.10020793913216]
  We present a new approach to novel view synthesis under time-varying illumination from such data. We introduce a new DeepMPI representation, motivated by observations on the sparsity structure of the plenoptic function. Our method can synthesize the same compelling parallax and view-dependent effects as previous MPI methods, while simultaneously interpolating along changes in reflectance and illumination with time.
  arXiv  Detail & Related papers  (2020-07-30T02:52:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.