Learning a Delighting Prior for Facial Appearance Capture in the Wild

• URL: http://arxiv.org/abs/2605.05636v1
• Date: Thu, 07 May 2026 03:43:24 GMT
• Title: Learning a Delighting Prior for Facial Appearance Capture in the Wild
• Abstract summary: We propose to shift the paradigm into training a powerful delighting network as a prior to constrain the optimization.<n>By conditioning the core network on learnable source-aware tokens, we decouple dataset-specific styles from physical delighting principles.<n>We transform the multi-view NeRSemble dataset into NeRSemble-Scan, a large-scale collection of 4K-resolution relightable scans.
• Score: 34.89149093909672
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: High-quality facial appearance capture has traditionally required costly studio recording. Recent works consider an in-the-wild smartphone-based setup; however, their model-based inverse rendering paradigm struggles with the complex disentanglement of reflectance from unknown illumination. To bridge this gap, we propose to shift the paradigm into training a powerful delighting network as a prior to constrain the optimization. We leverage the OLAT dataset and the rendered Light Stage scans for training, and propose Dataset Latent Modulation (DLM) to seamlessly integrate these heterogeneous data sources. Specifically, by conditioning the core network on learnable source-aware tokens, we decouple dataset-specific styles from physical delighting principles, enabling the emergence of a delighting prior that outperforms existing proprietary models. This powerful delighting prior enables a simple and automatic appearance capture pipeline that achieves high-quality reflectance estimation from casual video inputs, outperforming prior arts by a large margin. Furthermore, we leverage our appearance capture method to transform the multi-view NeRSemble dataset into NeRSemble-Scan, a large-scale collection of 4K-resolution relightable scans. By open-sourcing our model and the NeRSemble-Scan dataset, we democratize high-end facial capture and provide a new foundation for the research community to build photorealistic digital humans.

Related papers

• WildRelight: A Real-World Benchmark and Physics-Guided Adaptation for Single-Image Relighting [1.4145309506833206]
  We introduce WildRelight, the first in-the-wild dataset specifically created for evaluating single-image relighting models.<n>We show that the dataset allows synthetic models to align with real-world statistics on-the-fly, transforming the intractable sim-to-real challenge into a tractable self-supervised task.
  arXiv  Detail & Related papers  (2026-05-12T07:53:27Z)
• Leveraging Large-Scale Pretrained Spatial-Spectral Priors for General Zero-Shot Pansharpening [7.37033839561317]
  Existing deep learning methods for remote sensing image fusion often suffer from poor generalization when applied to unseen datasets.<n>We propose a novel pretraining strategy that leverages large-scale simulated datasets to learn robust spatial-spectral priors.<n>The pretrained models achieve superior results in zero-shot scenarios and show remarkable adaptation capability with minimal real data in one-shot settings.
  arXiv  Detail & Related papers  (2025-12-02T10:56:26Z)
• 3DPR: Single Image 3D Portrait Relight using Generative Priors [101.74130664920868]
  3DPR is an image-based relighting model that leverages generative priors learnt from multi-view One-Light-at-A-Time (OLAT) images.<n>We leverage the latent space of a pre-trained generative head model that provides a rich prior over face geometry learnt from in-the-wild image datasets.<n>Our reflectance network operates in the latent space of the generative head model, crucially enabling a relatively small number of lightstage images to train the reflectance model.
  arXiv  Detail & Related papers  (2025-10-17T17:37:42Z)
• Towards Realistic Low-Light Image Enhancement via ISP Driven Data Modeling [61.95831392879045]
  Deep neural networks (DNNs) have recently become the leading method for low-light image enhancement (LLIE)<n>Despite significant progress, their outputs may still exhibit issues such as amplified noise, incorrect white balance, or unnatural enhancements when deployed in real world applications.<n>A key challenge is the lack of diverse, large scale training data that captures the complexities of low-light conditions and imaging pipelines.<n>We propose a novel image signal processing (ISP) driven data synthesis pipeline that addresses these challenges by generating unlimited paired training data.
  arXiv  Detail & Related papers  (2025-04-16T15:53:53Z)
• SparseGS-W: Sparse-View 3D Gaussian Splatting in the Wild with Generative Priors [22.561786156613525]
  We propose SparseGS-W, a novel framework to Synthesizing novel views of large-scale scenes from unconstrained in-the-wild images.<n>We leverage geometric priors and constrained diffusion priors to compensate for the lack of multi-view information from extremely sparse input.<n>SparseGS-W achieves state-of-the-art performance not only in full-reference metrics, but also in commonly used non-reference metrics such as FID, ClipIQA, and MUSIQ.
  arXiv  Detail & Related papers  (2025-03-25T08:40:40Z)
• IDArb: Intrinsic Decomposition for Arbitrary Number of Input Views and Illuminations [64.07859467542664]
  Capturing geometric and material information from images remains a fundamental challenge in computer vision and graphics.<n>Traditional optimization-based methods often require hours of computational time to reconstruct geometry, material properties, and environmental lighting from dense multi-view inputs.<n>We introduce IDArb, a diffusion-based model designed to perform intrinsic decomposition on an arbitrary number of images under varying illuminations.
  arXiv  Detail & Related papers  (2024-12-16T18:52:56Z)
• LDM-ISP: Enhancing Neural ISP for Low Light with Latent Diffusion Models [54.93010869546011]
  We propose to leverage the pre-trained latent diffusion model to perform the neural ISP for enhancing extremely low-light images.<n>Specifically, to tailor the pre-trained latent diffusion model to operate on the RAW domain, we train a set of lightweight taming modules.<n>We observe different roles of UNet denoising and decoder reconstruction in the latent diffusion model, which inspires us to decompose the low-light image enhancement task into latent-space low-frequency content generation and decoding-phase high-frequency detail maintenance.
  arXiv  Detail & Related papers  (2023-12-02T04:31:51Z)
• Improving Lens Flare Removal with General Purpose Pipeline and Multiple Light Sources Recovery [69.71080926778413]
  flare artifacts can affect image visual quality and downstream computer vision tasks. Current methods do not consider automatic exposure and tone mapping in image signal processing pipeline. We propose a solution to improve the performance of lens flare removal by revisiting the ISP and design a more reliable light sources recovery strategy.
  arXiv  Detail & Related papers  (2023-08-31T04:58:17Z)
• UMat: Uncertainty-Aware Single Image High Resolution Material Capture [2.416160525187799]
  We propose a learning-based method to recover normals, specularity, and roughness from a single diffuse image of a material. Our method is the first one to deal with the problem of modeling uncertainty in material digitization.
  arXiv  Detail & Related papers  (2023-05-25T17:59:04Z)
• Energy-Inspired Self-Supervised Pretraining for Vision Models [36.70550531181131]
  We introduce a self-supervised vision model pretraining framework inspired by energy-based models (EBMs) In the proposed framework, we model energy estimation and data restoration as the forward and backward passes of a single network. We show the proposed method delivers comparable and even better performance with remarkably fewer epochs of training.
  arXiv  Detail & Related papers  (2023-02-02T19:41:00Z)
• Learning to Relight Portrait Images via a Virtual Light Stage and Synthetic-to-Real Adaptation [76.96499178502759]
  Relighting aims to re-illuminate the person in the image as if the person appeared in an environment with the target lighting. Recent methods rely on deep learning to achieve high-quality results. We propose a new approach that can perform on par with the state-of-the-art (SOTA) relighting methods without requiring a light stage.
  arXiv  Detail & Related papers  (2022-09-21T17:15:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.