Total-Editing: Head Avatar with Editable Appearance, Motion, and Lighting

• URL: http://arxiv.org/abs/2505.20582v1
• Date: Mon, 26 May 2025 23:32:51 GMT
• Title: Total-Editing: Head Avatar with Editable Appearance, Motion, and Lighting
• Authors: Yizhou Zhao, Chunjiang Liu, Haoyu Chen, Bhiksha Raj, Min Xu, Tadas Baltrusaitis, Mitch Rundle, HsiangTao Wu, Kamran Ghasedi,
• Abstract summary: Total-Editing is a unified portrait editing framework that enables precise control over appearance, motion, and lighting.<n>Specifically, we design a neural radiance decoder field with intrinsic decomposition capabilities.<n>We also incorporate a moving least squares based field to enhance thetemporal coherence of avatar motion and shading effects.
• Score: 39.44285683162209
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Face reenactment and portrait relighting are essential tasks in portrait editing, yet they are typically addressed independently, without much synergy. Most face reenactment methods prioritize motion control and multiview consistency, while portrait relighting focuses on adjusting shading effects. To take advantage of both geometric consistency and illumination awareness, we introduce Total-Editing, a unified portrait editing framework that enables precise control over appearance, motion, and lighting. Specifically, we design a neural radiance field decoder with intrinsic decomposition capabilities. This allows seamless integration of lighting information from portrait images or HDR environment maps into synthesized portraits. We also incorporate a moving least squares based deformation field to enhance the spatiotemporal coherence of avatar motion and shading effects. With these innovations, our unified framework significantly improves the quality and realism of portrait editing results. Further, the multi-source nature of Total-Editing supports more flexible applications, such as illumination transfer from one portrait to another, or portrait animation with customized backgrounds.

Related papers

• RASA: Replace Anyone, Say Anything -- A Training-Free Framework for Audio-Driven and Universal Portrait Video Editing [82.132107140504]
  We introduce a training-free universal portrait video editing framework that provides a versatile and adaptable editing strategy.<n>It supports portrait appearance editing conditioned on the changed first reference frame, as well as lip editing conditioned on varied speech.<n>Our model can achieve more accurate and synchronized lip movements for the lip editing task, as well as more flexible motion transfer for the appearance editing task.
  arXiv  Detail & Related papers  (2025-03-14T16:39:15Z)
• COMPOSE: Comprehensive Portrait Shadow Editing [25.727386174616868]
  COMPOSE is a novel shadow editing pipeline for human portraits. It offers precise control over shadow attributes such as shape, intensity, and position. We have trained models to: (1) predict this light source representation from images, and (2) generate realistic shadows using this representation.
  arXiv  Detail & Related papers  (2024-08-25T19:18:18Z)
• Lite2Relight: 3D-aware Single Image Portrait Relighting [87.62069509622226]
  Lite2Relight is a novel technique that can predict 3D consistent head poses of portraits. By utilizing a pre-trained geometry-aware encoder and a feature alignment module, we map input images into a relightable 3D space. This includes producing 3D-consistent results of the full head, including hair, eyes, and expressions.
  arXiv  Detail & Related papers  (2024-07-15T07:16:11Z)
• LightPainter: Interactive Portrait Relighting with Freehand Scribble [79.95574780974103]
  We introduce LightPainter, a scribble-based relighting system that allows users to interactively manipulate portrait lighting effect with ease. To train the relighting module, we propose a novel scribble simulation procedure to mimic real user scribbles. We demonstrate high-quality and flexible portrait lighting editing capability with both quantitative and qualitative experiments.
  arXiv  Detail & Related papers  (2023-03-22T23:17:11Z)
• NARRATE: A Normal Assisted Free-View Portrait Stylizer [42.38374601073052]
  NARRATE is a novel pipeline that enables simultaneously editing portrait lighting and perspective in a photorealistic manner. We experimentally demonstrate that NARRATE achieves more photorealistic, reliable results over prior works. We showcase vivid free-view facial animations as well as 3D-aware relightableization, which help facilitate various AR/VR applications.
  arXiv  Detail & Related papers  (2022-07-03T07:54:05Z)
• Hierarchical Vectorization for Portrait Images [12.32304366243904]
  We propose a novel vectorization method that can automatically convert images into a 3-tier hierarchical representation. The base layer consists of a set of sparse diffusion curves which characterize salient geometric features and low-frequency colors. The middle level encodes specular highlights and shadows to large and editable Poisson regions (PR) and allows the user to directly adjust illumination. The top level contains two types of pixel-sized PRs for high-frequency residuals and fine details such as pimples and pigmentation.
  arXiv  Detail & Related papers  (2022-05-24T07:58:41Z)
• Relightable Neural Video Portrait [36.67623086400362]
  Photo-realistic facial video portrait reenactment benefits virtual production and numerous VR/AR experiences. We present a relightable neural video portrait, a simultaneous relighting and reenactment scheme that transfers the head pose and facial expressions from a source actor to a portrait video of a target actor with arbitrary new backgrounds and lighting conditions.
  arXiv  Detail & Related papers  (2021-07-30T16:20:45Z)
• Pixel Sampling for Style Preserving Face Pose Editing [53.14006941396712]
  We present a novel two-stage approach to solve the dilemma, where the task of face pose manipulation is cast into face inpainting. By selectively sampling pixels from the input face and slightly adjust their relative locations, the face editing result faithfully keeps the identity information as well as the image style unchanged. With the 3D facial landmarks as guidance, our method is able to manipulate face pose in three degrees of freedom, i.e., yaw, pitch, and roll, resulting in more flexible face pose editing.
  arXiv  Detail & Related papers  (2021-06-14T11:29:29Z)
• PIE: Portrait Image Embedding for Semantic Control [82.69061225574774]
  We present the first approach for embedding real portrait images in the latent space of StyleGAN. We use StyleRig, a pretrained neural network that maps the control space of a 3D morphable face model to the latent space of the GAN. An identity energy preservation term allows spatially coherent edits while maintaining facial integrity.
  arXiv  Detail & Related papers  (2020-09-20T17:53:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.