Learning Latent Proxies for Controllable Single-Image Relighting

• URL: http://arxiv.org/abs/2603.15555v1
• Date: Mon, 16 Mar 2026 17:16:59 GMT
• Title: Learning Latent Proxies for Controllable Single-Image Relighting
• Abstract summary: Single-image relighting is highly under-constrained.<n>We introduce LightCtrl that integrates physical priors at two levels.<n>We also present ScaLight, a large-scale object-level dataset.
• Score: 29.76860042350802
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Single-image relighting is highly under-constrained: small illumination changes can produce large, nonlinear variations in shading, shadows, and specularities, while geometry and materials remain unobserved. Existing diffusion-based approaches either rely on intrinsic or G-buffer pipelines that require dense and fragile supervision, or operate purely in latent space without physical grounding, making fine-grained control of direction, intensity, and color unreliable. We observe that a full intrinsic decomposition is unnecessary and redundant for accurate relighting. Instead, sparse but physically meaningful cues, indicating where illumination should change and how materials should respond, are sufficient to guide a diffusion model. Based on this insight, we introduce LightCtrl that integrates physical priors at two levels: a few-shot latent proxy encoder that extracts compact material-geometry cues from limited PBR supervision, and a lighting-aware mask that identifies sensitive illumination regions and steers the denoiser toward shading relevant pixels. To compensate for scarce PBR data, we refine the proxy branch using a DPO-based objective that enforces physical consistency in the predicted cues. We also present ScaLight, a large-scale object-level dataset with systematically varied illumination and complete camera-light metadata, enabling physically consistent and controllable training. Across object and scene level benchmarks, our method achieves photometrically faithful relighting with accurate continuous control, surpassing prior diffusion and intrinsic-based baselines, including gains of up to +2.4 dB PSNR and 35% lower RMSE under controlled lighting shifts.

Related papers

• SpotlessGS: Relightable 3D Gaussian Splatting under Dynamic Illumination for Robotic Perception [20.515607443256073]
  We extend Dark Gaussian Splatting to a more accurate and flexible relightable 3D reconstruction framework.<n>We introduce a low-frequency illumination model based on spherical harmonics to capture spatially varying residual and ambient lighting effects.<n> Experiments on synthetic and real-world datasets demonstrate that our method effectively mitigates illumination artifacts while improving rendering quality and quantitative performance over prior approaches.
  arXiv  Detail & Related papers  (2026-08-11T17:35:34Z)
• FreeLit: Paired-Free Indoor Relighting via Physics-Guided Diffusion [12.521086833681077]
  FreeLit is a paired-free framework for controllable indoor relighting.<n>It explicitly manipulates light-source location, color, and intensity.<n>It achieves stable, physically consistent, and controllable relighting.
  arXiv  Detail & Related papers  (2026-07-15T10:03:22Z)
• Learning to Suppress SPAD-based LiDAR Flare [0.0]
  flare caused by excessive photon returns or pile-up effects can lead to incorrect depth estimation and exaggerated boundaries in point clouds.<n>Existing flare mitigation methods primarily operate at the hardware or signal-processing levels.<n>We propose Physically-Informed segmentation for LiDAR Flare (PILF), a learning-based approach that treats the first and second echoes.
  arXiv  Detail & Related papers  (2026-07-03T12:03:13Z)
• AIGS-Net: Compact Illumination Field Modeling via 2D Gaussian Splatting for Fast Low-Light Image Enhancement [19.712825613823778]
  This paper proposes an Adaptive Illumination Gaussian Splatting Network (AIGS-Net) for fast low-light enhancement.<n>Unlike conventional static priors, AIGS-Net constructs an input-adaptive 2D Gaussian Splatting illumination field.<n>Experiments show that AIGS-Net improves detail recovery and color fidelity while requiring only approximately 40 learnable parameters.
  arXiv  Detail & Related papers  (2026-06-16T14:45:55Z)
• SSD-GS: Scattering and Shadow Decomposition for Relightable 3D Gaussian Splatting [46.61006851229471]
  SSD-GS is a physically-based relighting framework built upon 3D Gaussian Splatting (3DGS)<n>It decomposes reflectance into four components: diffuse, specular, shadow, and subsurface scattering.<n> Experiments demonstrate superior quantitative and perceptual relighting quality compared to prior methods.
  arXiv  Detail & Related papers  (2026-04-14T22:47:04Z)
• LightMover: Generative Light Movement with Color and Intensity Controls [49.259165408623915]
  LightMover is a framework for controllable light manipulation in single images.<n>Video diffusion priors produce physically plausible illumination changes without re-rendering the scene.
  arXiv  Detail & Related papers  (2026-03-28T09:51:59Z)
• Diffusion-Based Low-Light Image Enhancement with Color and Luminance Priors [13.688097246812042]
  Low-light images often suffer from low contrast, noise, and color distortion, degrading visual quality and impairing downstream vision tasks.<n>We propose a novel conditional diffusion framework for low-light image enhancement that incorporates a Structured Control Embedding Module (SCEM)<n>SCEM decomposes a low-light image into four informative components including illumination, illumination-invariant features, shadow priors, and color-invariant cues.
  arXiv  Detail & Related papers  (2026-02-27T22:15:27Z)
• Unifying Color and Lightness Correction with View-Adaptive Curve Adjustment for Robust 3D Novel View Synthesis [73.27997579020233]
  We propose Luminance-GS++, a 3DGS-based framework for robust NVS under diverse illumination conditions.<n>Our method combines a globally view-adaptive lightness adjustment with a local pixel-wise residual refinement for precise color correction.
  arXiv  Detail & Related papers  (2026-02-20T16:20:50Z)
• POLAR: A Portrait OLAT Dataset and Generative Framework for Illumination-Aware Face Modeling [51.7495375918484]
  Face relighting aims to synthesize realistic portraits under novel illumination while preserving identity and geometry.<n>We introduce POLAR, a large-scale and physically calibrated One-Light-at-a-Time dataset containing over 200 subjects captured under 156 lighting directions, multiple views, and diverse expressions.<n>We develop a flow-based generative model POLARNet that predicts per-light OLAT responses from a single portrait, capturing fine-grained and direction-aware illumination effects while preserving facial identity.
  arXiv  Detail & Related papers  (2025-12-15T11:04:09Z)
• LightLab: Controlling Light Sources in Images with Diffusion Models [49.83835236202516]
  We present a diffusion-based method for fine-grained, parametric control over light sources in an image.<n>We leverage the linearity of light to synthesize image pairs depicting controlled light changes of either a target light source or ambient illumination.<n>We show how our method can achieve compelling light editing results, and outperforms existing methods based on user preference.
  arXiv  Detail & Related papers  (2025-05-14T17:57:27Z)
• DifFRelight: Diffusion-Based Facial Performance Relighting [12.909429637057343]
  We present a novel framework for free-viewpoint facial performance relighting using diffusion-based image-to-image translation. We train a diffusion model for precise lighting control, enabling high-fidelity relit facial images from flat-lit inputs. The model accurately reproduces complex lighting effects like eye reflections, subsurface scattering, self-shadowing, and translucency.
  arXiv  Detail & Related papers  (2024-10-10T17:56:44Z)
• Unsupervised Low-light Image Enhancement with Lookup Tables and Diffusion Priors [38.96909959677438]
  Low-light image enhancement (LIE) aims at precisely and efficiently recovering an image degraded in poor illumination environments. Recent advanced LIE techniques are using deep neural networks, which require lots of low-normal light image pairs, network parameters, and computational resources. We devise a novel unsupervised LIE framework based on diffusion priors and lookup tables to achieve efficient low-light image recovery.
  arXiv  Detail & Related papers  (2024-09-27T16:37:27Z)
• GS-ID: Illumination Decomposition on Gaussian Splatting via Adaptive Light Aggregation and Diffusion-Guided Material Priors [5.7153963416911]
  Gaussian Splatting (GS) has emerged as an effective representation for rendering, but the underlying geometry, material, and lighting remain entangled.<n>We propose textbfGS-ID, an end-to-end framework for illumination decomposition.<n> Experiments demonstrate the effectiveness of GS-ID for downstream applications such as relighting and scene composition.
  arXiv  Detail & Related papers  (2024-08-16T04:38:31Z)
• Improving Lens Flare Removal with General Purpose Pipeline and Multiple Light Sources Recovery [69.71080926778413]
  flare artifacts can affect image visual quality and downstream computer vision tasks. Current methods do not consider automatic exposure and tone mapping in image signal processing pipeline. We propose a solution to improve the performance of lens flare removal by revisiting the ISP and design a more reliable light sources recovery strategy.
  arXiv  Detail & Related papers  (2023-08-31T04:58:17Z)
• NeIF: Representing General Reflectance as Neural Intrinsics Fields for Uncalibrated Photometric Stereo [70.71400320657035]
  Uncalibrated photometric stereo (UPS) is challenging due to the inherent ambiguity brought by unknown light. This paper establishes an implicit relation between light clues and light estimation and solves UPS in an unsupervised manner.
  arXiv  Detail & Related papers  (2022-08-18T15:11:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.