LightHeadEd: Relightable & Editable Head Avatars from a Smartphone

• URL: http://arxiv.org/abs/2504.09671v1
• Date: Sun, 13 Apr 2025 17:51:56 GMT
• Title: LightHeadEd: Relightable & Editable Head Avatars from a Smartphone
• Authors: Pranav Manu, Astitva Srivastava, Amit Raj, Varun Jampani, Avinash Sharma, P. J. Narayanan,
• Abstract summary: We present a novel, cost-effective approach for creating high-quality relightable head avatars using only a smartphone equipped with polaroid filters.<n>Our approach involves simultaneously capturing cross-polarized and parallel-polarized video streams in a dark room with a single point-light source.<n>We introduce a hybrid representation that embeds 2D Gaussians in the UV space of a parametric head model, facilitating efficient real-time rendering while preserving high-fidelity geometric details.
• Score: 30.268643915885413
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Creating photorealistic, animatable, and relightable 3D head avatars traditionally requires expensive Lightstage with multiple calibrated cameras, making it inaccessible for widespread adoption. To bridge this gap, we present a novel, cost-effective approach for creating high-quality relightable head avatars using only a smartphone equipped with polaroid filters. Our approach involves simultaneously capturing cross-polarized and parallel-polarized video streams in a dark room with a single point-light source, separating the skin's diffuse and specular components during dynamic facial performances. We introduce a hybrid representation that embeds 2D Gaussians in the UV space of a parametric head model, facilitating efficient real-time rendering while preserving high-fidelity geometric details. Our learning-based neural analysis-by-synthesis pipeline decouples pose and expression-dependent geometrical offsets from appearance, decomposing the surface into albedo, normal, and specular UV texture maps, along with the environment maps. We collect a unique dataset of various subjects performing diverse facial expressions and head movements.

Related papers

• HRAvatar: High-Quality and Relightable Gaussian Head Avatar [34.274420686240866]
  We propose HRAvatar, a 3DGS-based method that reconstructs high-fidelity, relightable 3D head avatars.<n>It reduces tracking errors through end-to-end optimization and better captures individual facial deformations.<n>It decomposes head appearance into several physical properties and incorporates physically-based shading to account for environmental lighting.
  arXiv  Detail & Related papers  (2025-03-11T09:42:40Z)
• GaussianHeads: End-to-End Learning of Drivable Gaussian Head Avatars from Coarse-to-fine Representations [54.94362657501809]
  We propose a new method to generate highly dynamic and deformable human head avatars from multi-view imagery in real-time. At the core of our method is a hierarchical representation of head models that allows to capture the complex dynamics of facial expressions and head movements. We train this coarse-to-fine facial avatar model along with the head pose as a learnable parameter in an end-to-end framework.
  arXiv  Detail & Related papers  (2024-09-18T13:05:43Z)
• Interactive Rendering of Relightable and Animatable Gaussian Avatars [37.73483372890271]
  We propose a simple and efficient method to decouple body materials and lighting from multi-view or monocular avatar videos. Our method can render higher quality results at a faster speed on both synthetic and real datasets.
  arXiv  Detail & Related papers  (2024-07-15T13:25:07Z)
• Relightable Neural Actor with Intrinsic Decomposition and Pose Control [80.06094206522668]
  We propose Relightable Neural Actor, a new video-based method for learning a pose-driven neural human model that can be relighted. For training, our method solely requires a multi-view recording of the human under a known, but static lighting condition. To evaluate our approach in real-world scenarios, we collect a new dataset with four identities recorded under different light conditions, indoors and outdoors.
  arXiv  Detail & Related papers  (2023-12-18T14:30:13Z)
• Relightable Gaussian Codec Avatars [26.255161061306428]
  We present Relightable Gaussian Codec Avatars, a method to build high-fidelity relightable head avatars that can be animated to generate novel expressions. Our geometry model based on 3D Gaussians can capture 3D-consistent sub-millimeter details such as hair strands and pores on dynamic face sequences. We improve the fidelity of eye reflections and enable explicit gaze control by introducing relightable explicit eye models.
  arXiv  Detail & Related papers  (2023-12-06T18:59:58Z)
• FLARE: Fast Learning of Animatable and Relightable Mesh Avatars [64.48254296523977]
  Our goal is to efficiently learn personalized animatable 3D head avatars from videos that are geometrically accurate, realistic, relightable, and compatible with current rendering systems. We introduce FLARE, a technique that enables the creation of animatable and relightable avatars from a single monocular video.
  arXiv  Detail & Related papers  (2023-10-26T16:13:00Z)
• Spatiotemporally Consistent HDR Indoor Lighting Estimation [66.26786775252592]
  We propose a physically-motivated deep learning framework to solve the indoor lighting estimation problem. Given a single LDR image with a depth map, our method predicts spatially consistent lighting at any given image position. Our framework achieves photorealistic lighting prediction with higher quality compared to state-of-the-art single-image or video-based methods.
  arXiv  Detail & Related papers  (2023-05-07T20:36:29Z)
• HQ3DAvatar: High Quality Controllable 3D Head Avatar [65.70885416855782]
  This paper presents a novel approach to building highly photorealistic digital head avatars. Our method learns a canonical space via an implicit function parameterized by a neural network. At test time, our method is driven by a monocular RGB video.
  arXiv  Detail & Related papers  (2023-03-25T13:56:33Z)
• PointAvatar: Deformable Point-based Head Avatars from Videos [103.43941945044294]
  PointAvatar is a deformable point-based representation that disentangles the source color into intrinsic albedo and normal-dependent shading. We show that our method is able to generate animatable 3D avatars using monocular videos from multiple sources.
  arXiv  Detail & Related papers  (2022-12-16T10:05:31Z)
• High-Res Facial Appearance Capture from Polarized Smartphone Images [11.885559856133339]
  We propose a novel method for high-quality facial texture reconstruction from RGB images using a single smartphone. We record two short sequences in a dark environment under flash illumination with different light polarization using the modified smartphone. We then exploit the camera and light co-location within a differentiable structure to optimize the facial textures using an analysis-by-synthesis approach.
  arXiv  Detail & Related papers  (2022-12-02T13:34:56Z)
• Relightable 3D Head Portraits from a Smartphone Video [15.639140551193073]
  We present a system for creating a relightable 3D portrait of a human head. Our neural pipeline operates on a sequence of frames captured by a smartphone camera with the flash blinking. A deep rendering network is trained to regress dense albedo, normals, and environmental lighting maps for arbitrary new viewpoints.
  arXiv  Detail & Related papers  (2020-12-17T22:49:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.