Retinex-Diffusion: On Controlling Illumination Conditions in Diffusion Models via Retinex Theory

• URL: http://arxiv.org/abs/2407.20785v1
• Date: Mon, 29 Jul 2024 03:15:07 GMT
• Title: Retinex-Diffusion: On Controlling Illumination Conditions in Diffusion Models via Retinex Theory
• Authors: Xiaoyan Xing, Vincent Tao Hu, Jan Hendrik Metzen, Konrad Groh, Sezer Karaoglu, Theo Gevers,
• Abstract summary: We conceptualize the diffusion model as a black-box image render and strategically decompose its energy function in alignment with the image formation model. It generates images with realistic illumination effects, including cast shadow, soft shadow, and inter-reflections.
• Score: 19.205929427075965
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper introduces a novel approach to illumination manipulation in diffusion models, addressing the gap in conditional image generation with a focus on lighting conditions. We conceptualize the diffusion model as a black-box image render and strategically decompose its energy function in alignment with the image formation model. Our method effectively separates and controls illumination-related properties during the generative process. It generates images with realistic illumination effects, including cast shadow, soft shadow, and inter-reflections. Remarkably, it achieves this without the necessity for learning intrinsic decomposition, finding directions in latent space, or undergoing additional training with new datasets.

Related papers

• DifFRelight: Diffusion-Based Facial Performance Relighting [12.909429637057343]
  We present a novel framework for free-viewpoint facial performance relighting using diffusion-based image-to-image translation. We train a diffusion model for precise lighting control, enabling high-fidelity relit facial images from flat-lit inputs. The model accurately reproduces complex lighting effects like eye reflections, subsurface scattering, self-shadowing, and translucency.
  arXiv  Detail & Related papers  (2024-10-10T17:56:44Z)
• LightenDiffusion: Unsupervised Low-Light Image Enhancement with Latent-Retinex Diffusion Models [39.28266945709169]
  We propose a diffusion-based unsupervised framework that incorporates physically explainable Retinex theory with diffusion models for low-light image enhancement. Experiments on publicly available real-world benchmarks show that the proposed LightenDiffusion outperforms state-of-the-art unsupervised competitors.
  arXiv  Detail & Related papers  (2024-07-12T02:54:43Z)
• LightIt: Illumination Modeling and Control for Diffusion Models [61.80461416451116]
  We introduce LightIt, a method for explicit illumination control for image generation. Recent generative methods lack lighting control, which is crucial to numerous artistic aspects of image generation. Our method is the first that enables the generation of images with controllable, consistent lighting.
  arXiv  Detail & Related papers  (2024-03-15T18:26:33Z)
• Layered Rendering Diffusion Model for Zero-Shot Guided Image Synthesis [60.260724486834164]
  This paper introduces innovative solutions to enhance spatial controllability in diffusion models reliant on text queries. We present two key innovations: Vision Guidance and the Layered Rendering Diffusion framework. We apply our method to three practical applications: bounding box-to-image, semantic mask-to-image and image editing.
  arXiv  Detail & Related papers  (2023-11-30T10:36:19Z)
• Global Structure-Aware Diffusion Process for Low-Light Image Enhancement [64.69154776202694]
  This paper studies a diffusion-based framework to address the low-light image enhancement problem. We advocate for the regularization of its inherent ODE-trajectory. Experimental evaluations reveal that the proposed framework attains distinguished performance in low-light enhancement.
  arXiv  Detail & Related papers  (2023-10-26T17:01:52Z)
• Steered Diffusion: A Generalized Framework for Plug-and-Play Conditional Image Synthesis [62.07413805483241]
  Steered Diffusion is a framework for zero-shot conditional image generation using a diffusion model trained for unconditional generation. We present experiments using steered diffusion on several tasks including inpainting, colorization, text-guided semantic editing, and image super-resolution.
  arXiv  Detail & Related papers  (2023-09-30T02:03:22Z)
• LLDiffusion: Learning Degradation Representations in Diffusion Models for Low-Light Image Enhancement [118.83316133601319]
  Current deep learning methods for low-light image enhancement (LLIE) typically rely on pixel-wise mapping learned from paired data. We propose a degradation-aware learning scheme for LLIE using diffusion models, which effectively integrates degradation and image priors into the diffusion process.
  arXiv  Detail & Related papers  (2023-07-27T07:22:51Z)
• Image Harmonization with Diffusion Model [26.183879349798588]
  Inconsistent lighting conditions between the foreground and background often result in unrealistic composites. We present a novel approach for image harmonization by leveraging diffusion models.
  arXiv  Detail & Related papers  (2023-06-17T23:23:52Z)
• Diffusion in Diffusion: Cyclic One-Way Diffusion for Text-Vision-Conditioned Generation [11.80682025950519]
  In this work, we investigate the diffusion (physics) in diffusion (machine learning) properties. We propose our Cyclic One-Way Diffusion (COW) method to control the direction of diffusion phenomenon. Our method provides a novel perspective to understand the task needs and is applicable to a wider range of customization scenarios.
  arXiv  Detail & Related papers  (2023-06-14T05:25:06Z)
• Intrinsic Image Transfer for Illumination Manipulation [1.2387676601792899]
  This paper presents a novel intrinsic image transfer (IIT) algorithm for illumination manipulation. It creates a local image translation between two illumination surfaces. We illustrate that all losses can be reduced without the necessity of taking an intrinsic image decomposition.
  arXiv  Detail & Related papers  (2021-07-01T19:12:24Z)
• Learning Flow-based Feature Warping for Face Frontalization with Illumination Inconsistent Supervision [73.18554605744842]
  Flow-based Feature Warping Model (FFWM) learns to synthesize photo-realistic and illumination preserving frontal images. An Illumination Preserving Module (IPM) is proposed to learn illumination preserving image synthesis. A Warp Attention Module (WAM) is introduced to reduce the pose discrepancy in the feature level.
  arXiv  Detail & Related papers  (2020-08-16T06:07:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.