EgoRelight: Egocentric Human Capture and Illumination Recovery for Relightable and Photoreal Avatar Rendering

• URL: http://arxiv.org/abs/2605.28401v1
• Date: Wed, 27 May 2026 12:38:10 GMT
• Title: EgoRelight: Egocentric Human Capture and Illumination Recovery for Relightable and Photoreal Avatar Rendering
• Abstract summary: We present EgoRelight, a holistic framework for egocentric telepresence.<n>It simultaneously captures full-body human performance, synthesizes geometrics and relightable appearance.<n>We demonstrate our system through a social telepresence application, where remote users are coherently relit according to their physical environment.
• Score: 63.753771328356834
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mixed Reality (MR) headsets promise a future of immersive telepresence where virtual humans blend indistinguishably into real or virtual surroundings. Achieving this vision requires a method for capturing a user's motion, estimating appearance under novel lighting, and understanding the environment - all from the constrained viewpoint of a head-mounted display (HMD). Existing approaches treat these as isolated problems: they either focus on driving avatars with baked-in lighting or rely on studio setups for relighting. In this paper, we present EgoRelight, a holistic framework for egocentric telepresence that simultaneously captures full-body human performance, synthesizes photorealistic and relightable appearance, and estimates high dynamic range (HDR) environment maps from a single HMD. First, to ensure motion and surface reconstruction, we propose an egocentric perception module that leverages stereo down-facing cameras to extract dense depth maps, which serve as geometric control signals to drive a mesh-based avatar. Second, we introduce a novel neural appearance model that learns to synthesize view-dependent specular and view-independent diffuse shading separately. By employing a specialized ray-sampling strategy, our model generalizes to unseen illumination without relying on restrictive analytical BRDF priors. Third, we enable seamless avatar integration into the physical world via a test-time inverse rendering process, which recovers an HDR environment map by matching the pre-trained avatar's appearance to live egocentric camera observations. We demonstrate our system through a social telepresence application, where remote users are coherently relit according to their physical environment. Extensive experiments show that our components and the integrated system significantly outperform state-of-the-art baselines in geometric accuracy and rendering as well as relighting fidelity.

Related papers

• Generative Relightable Avatars [54.38436795769187]
  Generative Relightable Avatars (GRA) is a person-specific method for free-view rendering and environment-map relighting of full-body humans.<n>GRA follows a hybrid approach that combines controllable, physics-grounded relighting with probabilistic refinement.
  arXiv  Detail & Related papers  (2026-06-21T23:36:00Z)
• REFA: Real-time Egocentric Facial Animations for Virtual Reality [56.82169742343143]
  We present a novel system for real-time tracking of facial expressions using egocentric views captured from a set of infrared cameras embedded in a virtual reality (VR) headset.<n>Our technology facilitates any user to accurately drive the facial expressions of virtual characters in a non-intrusive manner.
  arXiv  Detail & Related papers  (2026-01-07T01:41:46Z)
• BecomingLit: Relightable Gaussian Avatars with Hybrid Neural Shading [3.447848701446988]
  We introduce BecomingLit, a novel method for reconstructing relightable, high-resolution head avatars that can be rendered from novel viewpoints at interactive rates.<n>We collect a novel dataset consisting of diverse multi-view sequences of numerous subjects under varying illumination conditions.<n>We propose a new hybrid neural shading approach, combining a neural diffuse BRDF with an analytical specular term.
  arXiv  Detail & Related papers  (2025-06-06T17:53:58Z)
• LightHeadEd: Relightable & Editable Head Avatars from a Smartphone [30.268643915885413]
  We present a novel, cost-effective approach for creating high-quality relightable head avatars using only a smartphone equipped with polaroid filters.<n>Our approach involves simultaneously capturing cross-polarized and parallel-polarized video streams in a dark room with a single point-light source.<n>We introduce a hybrid representation that embeds 2D Gaussians in the UV space of a parametric head model, facilitating efficient real-time rendering while preserving high-fidelity geometric details.
  arXiv  Detail & Related papers  (2025-04-13T17:51:56Z)
• URAvatar: Universal Relightable Gaussian Codec Avatars [42.25313535192927]
  We present a new approach to creating photorealistic and relightable head avatars from a phone scan with unknown illumination. The reconstructed avatars can be animated and relit in real time with the global illumination of diverse environments.
  arXiv  Detail & Related papers  (2024-10-31T17:59:56Z)
• EgoAvatar: Egocentric View-Driven and Photorealistic Full-body Avatars [56.56236652774294]
  We propose a person-specific egocentric telepresence approach, which jointly models the photoreal digital avatar while also driving it from a single egocentric video. Our experiments demonstrate a clear step towards egocentric and photoreal telepresence as our method outperforms baselines as well as competing methods.
  arXiv  Detail & Related papers  (2024-09-22T22:50:27Z)
• FLARE: Fast Learning of Animatable and Relightable Mesh Avatars [64.48254296523977]
  Our goal is to efficiently learn personalized animatable 3D head avatars from videos that are geometrically accurate, realistic, relightable, and compatible with current rendering systems. We introduce FLARE, a technique that enables the creation of animatable and relightable avatars from a single monocular video.
  arXiv  Detail & Related papers  (2023-10-26T16:13:00Z)
• EgoRenderer: Rendering Human Avatars from Egocentric Camera Images [87.96474006263692]
  We present EgoRenderer, a system for rendering full-body neural avatars of a person captured by a wearable, egocentric fisheye camera. Rendering full-body avatars from such egocentric images come with unique challenges due to the top-down view and large distortions. We tackle these challenges by decomposing the rendering process into several steps, including texture synthesis, pose construction, and neural image translation.
  arXiv  Detail & Related papers  (2021-11-24T18:33:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.