WildCap: Facial Appearance Capture in the Wild via Hybrid Inverse Rendering

• URL: http://arxiv.org/abs/2512.11237v1
• Date: Fri, 12 Dec 2025 02:37:03 GMT
• Title: WildCap: Facial Appearance Capture in the Wild via Hybrid Inverse Rendering
• Authors: Yuxuan Han, Xin Ming, Tianxiao Li, Zhuofan Shen, Qixuan Zhang, Lan Xu, Feng Xu,
• Abstract summary: Existing methods achieve high-quality facial appearance capture under controllable lighting.<n>We propose WildCap, a novel method for high-quality facial appearance capture from a smartphone video recorded in the wild.
• Score: 34.89149093909672
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing methods achieve high-quality facial appearance capture under controllable lighting, which increases capture cost and limits usability. We propose WildCap, a novel method for high-quality facial appearance capture from a smartphone video recorded in the wild. To disentangle high-quality reflectance from complex lighting effects in in-the-wild captures, we propose a novel hybrid inverse rendering framework. Specifically, we first apply a data-driven method, i.e., SwitchLight, to convert the captured images into more constrained conditions and then adopt model-based inverse rendering. However, unavoidable local artifacts in network predictions, such as shadow-baking, are non-physical and thus hinder accurate inverse rendering of lighting and material. To address this, we propose a novel texel grid lighting model to explain non-physical effects as clean albedo illuminated by local physical lighting. During optimization, we jointly sample a diffusion prior for reflectance maps and optimize the lighting, effectively resolving scale ambiguity between local lights and albedo. Our method achieves significantly better results than prior arts in the same capture setup, closing the quality gap between in-the-wild and controllable recordings by a large margin. Our code will be released \href{https://yxuhan.github.io/WildCap/index.html}{\textcolor{magenta}{here}}.

Related papers

• SynthLight: Portrait Relighting with Diffusion Model by Learning to Re-render Synthetic Faces [16.65498750779018]
  We introduce SynthLight, a diffusion model for portrait relighting.<n>Our approach frames image relighting as a re-rendering problem, where pixels are transformed in response to changes in environmental lighting conditions.<n>We synthesize a dataset to simulate this lighting-conditioned transformation with 3D head assets under varying lighting.
  arXiv  Detail & Related papers  (2025-01-16T18:59:48Z)
• Dimma: Semi-supervised Low Light Image Enhancement with Adaptive Dimming [0.728258471592763]
  Enhancing low-light images while maintaining natural colors is a challenging problem due to camera processing variations. We propose Dimma, a semi-supervised approach that aligns with any camera by utilizing a small set of image pairs. We achieve that by introducing a convolutional mixture density network that generates distorted colors of the scene based on the illumination differences.
  arXiv  Detail & Related papers  (2023-10-14T17:59:46Z)
• WildLight: In-the-wild Inverse Rendering with a Flashlight [77.31815397135381]
  We propose a practical photometric solution for in-the-wild inverse rendering under unknown ambient lighting. Our system recovers scene geometry and reflectance using only multi-view images captured by a smartphone. We demonstrate by extensive experiments that our method is easy to implement, casual to set up, and consistently outperforms existing in-the-wild inverse rendering techniques.
  arXiv  Detail & Related papers  (2023-03-24T17:59:56Z)
• Learning to Relight Portrait Images via a Virtual Light Stage and Synthetic-to-Real Adaptation [76.96499178502759]
  Relighting aims to re-illuminate the person in the image as if the person appeared in an environment with the target lighting. Recent methods rely on deep learning to achieve high-quality results. We propose a new approach that can perform on par with the state-of-the-art (SOTA) relighting methods without requiring a light stage.
  arXiv  Detail & Related papers  (2022-09-21T17:15:58Z)
• Geometry-aware Single-image Full-body Human Relighting [37.381122678376805]
  Single-image human relighting aims to relight a target human under new lighting conditions by decomposing the input image into albedo, shape and lighting. Previous methods suffer from both the entanglement between albedo and lighting and the lack of hard shadows. Our framework is able to generate photo-realistic high-frequency shadows such as cast shadows under challenging lighting conditions.
  arXiv  Detail & Related papers  (2022-07-11T10:21:02Z)
• SAMURAI: Shape And Material from Unconstrained Real-world Arbitrary Image collections [49.3480550339732]
  Inverse rendering of an object under entirely unknown capture conditions is a fundamental challenge in computer vision and graphics. We propose a joint optimization framework to estimate the shape, BRDF, and per-image camera pose and illumination. Our method works on in-the-wild online image collections of an object and produces relightable 3D assets for several use-cases such as AR/VR.
  arXiv  Detail & Related papers  (2022-05-31T13:16:48Z)
• Neural Radiance Fields for Outdoor Scene Relighting [70.97747511934705]
  We present NeRF-OSR, the first approach for outdoor scene relighting based on neural radiance fields. In contrast to the prior art, our technique allows simultaneous editing of both scene illumination and camera viewpoint. It also includes a dedicated network for shadow reproduction, which is crucial for high-quality outdoor scene relighting.
  arXiv  Detail & Related papers  (2021-12-09T18:59:56Z)
• Low-light Image Enhancement via Breaking Down the Darkness [8.707025631892202]
  This paper presents a novel framework inspired by the divide-and-rule principle. We propose to convert an image from the RGB space into a luminance-chrominance one. An adjustable noise suppression network is designed to eliminate noise in the brightened luminance. The enhanced luminance further serves as guidance for the chrominance mapper to generate realistic colors.
  arXiv  Detail & Related papers  (2021-11-30T16:50:59Z)
• Relighting Images in the Wild with a Self-Supervised Siamese Auto-Encoder [62.580345486483886]
  We propose a self-supervised method for image relighting of single view images in the wild. The method is based on an auto-encoder which deconstructs an image into two separate encodings. We train our model on large-scale datasets such as Youtube 8M and CelebA.
  arXiv  Detail & Related papers  (2020-12-11T16:08:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.