MixLight: Borrowing the Best of both Spherical Harmonics and Gaussian Models

• URL: http://arxiv.org/abs/2404.12768v1
• Date: Fri, 19 Apr 2024 10:17:10 GMT
• Title: MixLight: Borrowing the Best of both Spherical Harmonics and Gaussian Models
• Authors: Xinlong Ji, Fangneng Zhan, Shijian Lu, Shi-Sheng Huang, Hua Huang,
• Abstract summary: Existing works estimate illumination by generating illumination maps or regressing illumination parameters. This paper presents MixLight, a joint model that utilizes the complementary characteristics of SH and SG to achieve a more complete illumination representation.
• Score: 69.39388799906409
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Accurately estimating scene lighting is critical for applications such as mixed reality. Existing works estimate illumination by generating illumination maps or regressing illumination parameters. However, the method of generating illumination maps has poor generalization performance and parametric models such as Spherical Harmonic (SH) and Spherical Gaussian (SG) fall short in capturing high-frequency or low-frequency components. This paper presents MixLight, a joint model that utilizes the complementary characteristics of SH and SG to achieve a more complete illumination representation, which uses SH and SG to capture low-frequency ambient and high-frequency light sources respectively. In addition, a special spherical light source sparsemax (SLSparsemax) module that refers to the position and brightness relationship between spherical light sources is designed to improve their sparsity, which is significant but omitted by prior works. Extensive experiments demonstrate that MixLight surpasses state-of-the-art (SOTA) methods on multiple metrics. In addition, experiments on Web Dataset also show that MixLight as a parametric method has better generalization performance than non-parametric methods.

Related papers

• LIPIDS: Learning-based Illumination Planning In Discretized (Light) Space for Photometric Stereo [19.021200954913475]
  Photometric stereo is a powerful method for obtaining per-pixel surface normals from differently illuminated images of an object. Finding an optimal configuration is challenging due to the vast number of possible lighting directions. We introduce LIPIDS - Learning-based Illumination Planning In Discretized light Space.
  arXiv  Detail & Related papers  (2024-09-01T09:54:16Z)
• GS-Phong: Meta-Learned 3D Gaussians for Relightable Novel View Synthesis [63.5925701087252]
  We propose a novel method for representing a scene illuminated by a point light using a set of relightable 3D Gaussian points. Inspired by the Blinn-Phong model, our approach decomposes the scene into ambient, diffuse, and specular components. To facilitate the decomposition of geometric information independent of lighting conditions, we introduce a novel bilevel optimization-based meta-learning framework.
  arXiv  Detail & Related papers  (2024-05-31T13:48:54Z)
• NeFII: Inverse Rendering for Reflectance Decomposition with Near-Field Indirect Illumination [48.42173911185454]
  Inverse rendering methods aim to estimate geometry, materials and illumination from multi-view RGB images. We propose an end-to-end inverse rendering pipeline that decomposes materials and illumination from multi-view images.
  arXiv  Detail & Related papers  (2023-03-29T12:05:19Z)
• Sparse Needlets for Lighting Estimation with Spherical Transport Loss [89.52531416604774]
  NeedleLight is a new lighting estimation model that represents illumination with needlets and allows lighting estimation in both frequency domain and spatial domain jointly. Extensive experiments show that NeedleLight achieves superior lighting estimation consistently across multiple evaluation metrics as compared with state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-06-24T15:19:42Z)
• Leveraging Spatial and Photometric Context for Calibrated Non-Lambertian Photometric Stereo [61.6260594326246]
  We introduce an efficient fully-convolutional architecture that can leverage both spatial and photometric context simultaneously. Using separable 4D convolutions and 2D heat-maps reduces the size and makes more efficient.
  arXiv  Detail & Related papers  (2021-03-22T18:06:58Z)
• GMLight: Lighting Estimation via Geometric Distribution Approximation [86.95367898017358]
  This paper presents a lighting estimation framework that employs a regression network and a generative projector for effective illumination estimation. We parameterize illumination scenes in terms of the geometric light distribution, light intensity, ambient term, and auxiliary depth, and estimate them as a pure regression task. With the estimated lighting parameters, the generative projector synthesizes panoramic illumination maps with realistic appearance and frequency.
  arXiv  Detail & Related papers  (2021-02-20T03:31:52Z)
• EMLight: Lighting Estimation via Spherical Distribution Approximation [33.26530733479459]
  We propose an illumination estimation framework that leverages a regression network and a neural projector for accurate illumination estimation. We decompose the illumination map into spherical light distribution, light intensity and the ambient term. Under the guidance of the predicted spherical distribution, light intensity and ambient term, the neural projector synthesizes panoramic illumination maps with realistic light frequency.
  arXiv  Detail & Related papers  (2020-12-21T04:54:08Z)
• PointAR: Efficient Lighting Estimation for Mobile Augmented Reality [7.58114840374767]
  We propose an efficient lighting estimation pipeline that is suitable to run on modern mobile devices. PointAR takes a single RGB-D image captured from the mobile camera and a 2D location in that image, and estimates 2nd order spherical harmonics coefficients.
  arXiv  Detail & Related papers  (2020-03-30T19:13:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.