Local Relighting of Real Scenes
- URL: http://arxiv.org/abs/2207.02774v1
- Date: Wed, 6 Jul 2022 16:08:20 GMT
- Title: Local Relighting of Real Scenes
- Authors: Audrey Cui, Ali Jahanian, Agata Lapedriza, Antonio Torralba, Shahin
Mahdizadehaghdam, Rohit Kumar, David Bau
- Abstract summary: We introduce the task of local relighting, which changes a photograph of a scene by switching on and off the light sources that are visible within the image.
This new task differs from the traditional image relighting problem, as it introduces the challenge of detecting light sources and inferring the pattern of light that emanates from them.
We propose an approach for local relighting that trains a model without supervision of any novel image dataset by using synthetically generated image pairs from another model.
- Score: 31.305393724281604
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We introduce the task of local relighting, which changes a photograph of a
scene by switching on and off the light sources that are visible within the
image. This new task differs from the traditional image relighting problem, as
it introduces the challenge of detecting light sources and inferring the
pattern of light that emanates from them. We propose an approach for local
relighting that trains a model without supervision of any novel image dataset
by using synthetically generated image pairs from another model. Concretely, we
collect paired training images from a stylespace-manipulated GAN; then we use
these images to train a conditional image-to-image model. To benchmark local
relighting, we introduce Lonoff, a collection of 306 precisely aligned images
taken in indoor spaces with different combinations of lights switched on. We
show that our method significantly outperforms baseline methods based on GAN
inversion. Finally, we demonstrate extensions of our method that control
different light sources separately. We invite the community to tackle this new
task of local relighting.
Related papers
- Relightful Harmonization: Lighting-aware Portrait Background Replacement [23.19641174787912]
We introduce Relightful Harmonization, a lighting-aware diffusion model designed to seamlessly harmonize sophisticated lighting effect for the foreground portrait using any background image.
Our approach unfolds in three stages. First, we introduce a lighting representation module that allows our diffusion model to encode lighting information from target image background.
Second, we introduce an alignment network that aligns lighting features learned from image background with lighting features learned from panorama environment maps.
arXiv Detail & Related papers (2023-12-11T23:20:31Z) - Neural Radiance Transfer Fields for Relightable Novel-view Synthesis
with Global Illumination [63.992213016011235]
We propose a method for scene relighting under novel views by learning a neural precomputed radiance transfer function.
Our method can be solely supervised on a set of real images of the scene under a single unknown lighting condition.
Results show that the recovered disentanglement of scene parameters improves significantly over the current state of the art.
arXiv Detail & Related papers (2022-07-27T16:07:48Z) - Enhancing Low-Light Images in Real World via Cross-Image Disentanglement [58.754943762945864]
We propose a new low-light image enhancement dataset consisting of misaligned training images with real-world corruptions.
Our model achieves state-of-the-art performances on both the newly proposed dataset and other popular low-light datasets.
arXiv Detail & Related papers (2022-01-10T03:12:52Z) - SILT: Self-supervised Lighting Transfer Using Implicit Image
Decomposition [27.72518108918135]
The solution operates as a two-branch network that first aims to map input images of any arbitrary lighting style to a unified domain.
We then remap this unified input domain using a discriminator that is presented with the generated outputs and the style reference.
Our method is shown to outperform supervised relighting solutions across two different datasets without requiring lighting supervision.
arXiv Detail & Related papers (2021-10-25T12:52:53Z) - Enhance Images as You Like with Unpaired Learning [8.104571453311442]
We propose a lightweight one-path conditional generative adversarial network (cGAN) to learn a one-to-many relation from low-light to normal-light image space.
Our network learns to generate a collection of enhanced images from a given input conditioned on various reference images.
Our model achieves competitive visual and quantitative results on par with fully supervised methods on both noisy and clean datasets.
arXiv Detail & Related papers (2021-10-04T03:00:44Z) - Relighting Images in the Wild with a Self-Supervised Siamese
Auto-Encoder [62.580345486483886]
We propose a self-supervised method for image relighting of single view images in the wild.
The method is based on an auto-encoder which deconstructs an image into two separate encodings.
We train our model on large-scale datasets such as Youtube 8M and CelebA.
arXiv Detail & Related papers (2020-12-11T16:08:50Z) - Light Stage Super-Resolution: Continuous High-Frequency Relighting [58.09243542908402]
We propose a learning-based solution for the "super-resolution" of scans of human faces taken from a light stage.
Our method aggregates the captured images corresponding to neighboring lights in the stage, and uses a neural network to synthesize a rendering of the face.
Our learned model is able to produce renderings for arbitrary light directions that exhibit realistic shadows and specular highlights.
arXiv Detail & Related papers (2020-10-17T23:40:43Z) - Crowdsampling the Plenoptic Function [56.10020793913216]
We present a new approach to novel view synthesis under time-varying illumination from such data.
We introduce a new DeepMPI representation, motivated by observations on the sparsity structure of the plenoptic function.
Our method can synthesize the same compelling parallax and view-dependent effects as previous MPI methods, while simultaneously interpolating along changes in reflectance and illumination with time.
arXiv Detail & Related papers (2020-07-30T02:52:10Z) - Scene relighting with illumination estimation in the latent space on an
encoder-decoder scheme [68.8204255655161]
In this report we present methods that we tried to achieve that goal.
Our models are trained on a rendered dataset of artificial locations with varied scene content, light source location and color temperature.
With this dataset, we used a network with illumination estimation component aiming to infer and replace light conditions in the latent space representation of the concerned scenes.
arXiv Detail & Related papers (2020-06-03T15:25:11Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.