LumiNet: Latent Intrinsics Meets Diffusion Models for Indoor Scene Relighting

• URL: http://arxiv.org/abs/2412.00177v2
• Date: Tue, 03 Dec 2024 17:21:41 GMT
• Title: LumiNet: Latent Intrinsics Meets Diffusion Models for Indoor Scene Relighting
• Authors: Xiaoyan Xing, Konrad Groh, Sezer Karaoglu, Theo Gevers, Anand Bhattad,
• Abstract summary: Given a source image and a target lighting image, LumiNet synthesizes a relit version of the source scene that captures the target's lighting. LumiNet processes latent representations from two different images - preserving geometry and albedo from the source while transferring lighting characteristics from the target.
• Score: 13.433775723052753
• License:
• Abstract: We introduce LumiNet, a novel architecture that leverages generative models and latent intrinsic representations for effective lighting transfer. Given a source image and a target lighting image, LumiNet synthesizes a relit version of the source scene that captures the target's lighting. Our approach makes two key contributions: a data curation strategy from the StyleGAN-based relighting model for our training, and a modified diffusion-based ControlNet that processes both latent intrinsic properties from the source image and latent extrinsic properties from the target image. We further improve lighting transfer through a learned adaptor (MLP) that injects the target's latent extrinsic properties via cross-attention and fine-tuning. Unlike traditional ControlNet, which generates images with conditional maps from a single scene, LumiNet processes latent representations from two different images - preserving geometry and albedo from the source while transferring lighting characteristics from the target. Experiments demonstrate that our method successfully transfers complex lighting phenomena including specular highlights and indirect illumination across scenes with varying spatial layouts and materials, outperforming existing approaches on challenging indoor scenes using only images as input.

Related papers

• Materialist: Physically Based Editing Using Single-Image Inverse Rendering [50.39048790589746]
  We present a method combining a learning-based approach with progressive differentiable rendering. Our method achieves more realistic light material interactions, accurate shadows, and global illumination. We also propose a method for material transparency editing that operates effectively without requiring full scene geometry.
  arXiv  Detail & Related papers  (2025-01-07T11:52:01Z)
• LumiSculpt: A Consistency Lighting Control Network for Video Generation [67.48791242688493]
  Lighting plays a pivotal role in ensuring the naturalness of video generation. It remains challenging to disentangle and model independent and coherent lighting attributes. LumiSculpt enables precise and consistent lighting control in T2V generation models.
  arXiv  Detail & Related papers  (2024-10-30T12:44:08Z)
• DifFRelight: Diffusion-Based Facial Performance Relighting [12.909429637057343]
  We present a novel framework for free-viewpoint facial performance relighting using diffusion-based image-to-image translation. We train a diffusion model for precise lighting control, enabling high-fidelity relit facial images from flat-lit inputs. The model accurately reproduces complex lighting effects like eye reflections, subsurface scattering, self-shadowing, and translucency.
  arXiv  Detail & Related papers  (2024-10-10T17:56:44Z)
• Retinex-Diffusion: On Controlling Illumination Conditions in Diffusion Models via Retinex Theory [19.205929427075965]
  We conceptualize the diffusion model as a black-box image render and strategically decompose its energy function in alignment with the image formation model. It generates images with realistic illumination effects, including cast shadow, soft shadow, and inter-reflections.
  arXiv  Detail & Related papers  (2024-07-29T03:15:07Z)
• Designing An Illumination-Aware Network for Deep Image Relighting [69.750906769976]
  We present an Illumination-Aware Network (IAN) which follows the guidance from hierarchical sampling to progressively relight a scene from a single image. In addition, an Illumination-Aware Residual Block (IARB) is designed to approximate the physical rendering process. Experimental results show that our proposed method produces better quantitative and qualitative relighting results than previous state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-07-21T16:21:24Z)
• Local Relighting of Real Scenes [31.305393724281604]
  We introduce the task of local relighting, which changes a photograph of a scene by switching on and off the light sources that are visible within the image. This new task differs from the traditional image relighting problem, as it introduces the challenge of detecting light sources and inferring the pattern of light that emanates from them. We propose an approach for local relighting that trains a model without supervision of any novel image dataset by using synthetically generated image pairs from another model.
  arXiv  Detail & Related papers  (2022-07-06T16:08:20Z)
• StyLitGAN: Prompting StyleGAN to Produce New Illumination Conditions [1.933681537640272]
  We propose a novel method, StyLitGAN, for relighting and resurfacing generated images in the absence of labeled data. Our approach generates images with realistic lighting effects, including cast shadows, soft shadows, inter-reflections, and glossy effects, without the need for paired or CGI data.
  arXiv  Detail & Related papers  (2022-05-20T17:59:40Z)
• Physically-Based Editing of Indoor Scene Lighting from a Single Image [106.60252793395104]
  We present a method to edit complex indoor lighting from a single image with its predicted depth and light source segmentation masks. We tackle this problem using two novel components: 1) a holistic scene reconstruction method that estimates scene reflectance and parametric 3D lighting, and 2) a neural rendering framework that re-renders the scene from our predictions.
  arXiv  Detail & Related papers  (2022-05-19T06:44:37Z)
• DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderer [78.91753256634453]
  We consider the challenging problem of predicting intrinsic object properties from a single image by exploiting differentiables. In this work, we propose DIBR++, a hybrid differentiable which supports these effects by combining specularization and ray-tracing. Compared to more advanced physics-based differentiables, DIBR++ is highly performant due to its compact and expressive model.
  arXiv  Detail & Related papers  (2021-10-30T01:59:39Z)
• Physically Inspired Dense Fusion Networks for Relighting [45.66699760138863]
  We propose a model which enriches neural networks with physical insight. Our method generates the relighted image with new illumination settings via two different strategies. We show that our proposal can outperform many state-of-the-art methods in terms of well-known fidelity metrics and perceptual loss.
  arXiv  Detail & Related papers  (2021-05-05T17:33:45Z)
• Light Stage Super-Resolution: Continuous High-Frequency Relighting [58.09243542908402]
  We propose a learning-based solution for the "super-resolution" of scans of human faces taken from a light stage. Our method aggregates the captured images corresponding to neighboring lights in the stage, and uses a neural network to synthesize a rendering of the face. Our learned model is able to produce renderings for arbitrary light directions that exhibit realistic shadows and specular highlights.
  arXiv  Detail & Related papers  (2020-10-17T23:40:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.