Photorealistic Novel View Synthesis of Human Faces using Next-Scale Transformers

• URL: http://arxiv.org/abs/2608.23410v1
• Date: Mon, 24 Aug 2026 15:55:34 GMT
• Title: Photorealistic Novel View Synthesis of Human Faces using Next-Scale Transformers
• Abstract summary: We build on the next-scale autoregressive paradigm and adapt it for human-centric view synthesis.<n>We train on a synthetic dataset of human faces spanning diverse identities and apparel.<n>The resulting model produces sharp and realistic views, with the option to synthesize multiple novel viewpoints simultaneously.
• Score: 8.260375854707354
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Photorealistic novel view synthesis of people remains challenging at high spatial resolutions and across multiple target cameras, where preserving identity, fine appearance details, and geometric coherence is critical. We build on the next-scale autoregressive paradigm and adapt it for human-centric view synthesis by enabling higher image resolutions, multi-view outputs and stronger cross-view consistency in a single forward pass. We train on a synthetic dataset of human faces spanning diverse identities and apparel. Contrary to diffusion models, this paradigm does not need 2D pre-training and, thanks to its next-scale architecture, it benefits from lower-resolution, general-purpose pre-trainings, with the full-sized purpose-specific images being used only in the last training stages. This enables our architecture to converge with a smaller amount of purpose-specific training data, allowing us to use a smaller but more realistic training dataset. The resulting model produces sharp and realistic views, with the option to synthesize multiple novel viewpoints simultaneously for improved agreement across views. Empirically, we observe gains in perceptual fidelity and cross-view coherence on human subjects, demonstrating that next-scale autoregression is an effective backbone for scalable, multi-output human view synthesis. We also couple our pipeline with an existing transformer-based model for pixel-aligned 3D gaussian lifting from multi-view facial inputs, resulting in accurate and photorealistic 3D models of human faces.

Related papers

• HumanNOVA: Photorealistic, Universal and Rapid 3D Human Avatar Modeling from a Single Image [84.81016200801153]
  We present HumanNOVA, a photorealistic, universal, and rapid model for generating 3D human avatars from a single RGB image.<n>In terms of the architecture, HumanNOVA adopts a feed-forward, token-conditioned avatar modeling framework that allows fast inference in less than one second.
  arXiv  Detail & Related papers  (2026-06-01T17:58:11Z)
• HumanRAM: Feed-forward Human Reconstruction and Animation Model using Transformers [60.86393841247567]
  HumanRAM is a novel feed-forward approach for generalizable human reconstruction and animation from monocular or sparse human images.<n>Our approach integrates human reconstruction and animation into a unified framework by introducing explicit pose conditions.<n> Experiments show that HumanRAM significantly surpasses previous methods in terms of reconstruction accuracy, animation fidelity, and generalization performance on real-world datasets.
  arXiv  Detail & Related papers  (2025-06-03T17:50:05Z)
• GAS: Generative Avatar Synthesis from a Single Image [54.95198111659466]
  We present a framework for synthesizing view-consistent and temporally coherent avatars from a single image.<n>Our approach combines the reconstruction power of regression-based 3D human reconstruction with the generative capabilities of a diffusion model.
  arXiv  Detail & Related papers  (2025-02-10T19:00:39Z)
• Synthetic Prior for Few-Shot Drivable Head Avatar Inversion [61.51887011274453]
  We present SynShot, a novel method for the few-shot inversion of a drivable head avatar based on a synthetic prior.<n>Inspired by machine learning models trained solely on synthetic data, we propose a method that learns a prior model from a large dataset of synthetic heads.
  arXiv  Detail & Related papers  (2025-01-12T19:01:05Z)
• FaceLift: Learning Generalizable Single Image 3D Face Reconstruction from Synthetic Heads [54.24070918942727]
  We present FaceLift, a novel feed-forward approach for high-quality 360-degree 3D head reconstruction from a single image.<n>Our pipeline first employs a multi-view latent diffusion model to generate consistent side and back views from a single input.<n>We show that FaceLift outperforms state-of-the-art 3D face reconstruction methods on identity preservation, detail recovery, and rendering quality.
  arXiv  Detail & Related papers  (2024-12-23T18:59:49Z)
• Drive-1-to-3: Enriching Diffusion Priors for Novel View Synthesis of Real Vehicles [81.29018359825872]
  This paper consolidates a set of good practices to finetune large pretrained models for a real-world task.<n>Specifically, we develop several strategies to account for discrepancies between the synthetic data and real driving data.<n>Our insights lead to effective finetuning that results in a $68.8%$ reduction in FID for novel view synthesis over prior arts.
  arXiv  Detail & Related papers  (2024-12-19T03:39:13Z)
• Human Multi-View Synthesis from a Single-View Model:Transferred Body and Face Representations [7.448124739584319]
  We propose an innovative framework that leverages transferred body and facial representations for multi-view human synthesis.<n> Specifically, we use a single-view model pretrained on a large-scale human dataset to develop a multi-view body representation.<n>Our approach outperforms the current state-of-the-art methods, achieving superior performance in multi-view human synthesis.
  arXiv  Detail & Related papers  (2024-12-04T04:02:17Z)
• Diffusion Models are Efficient Data Generators for Human Mesh Recovery [55.37787289869703]
  We show that synthetic data created by generative models is complementary to CG-rendered data.<n>We propose an effective data generation pipeline based on recent diffusion models, termed HumanWild.<n>Our work could pave the way for scaling up 3D human recovery to in-the-wild scenes.
  arXiv  Detail & Related papers  (2024-03-17T06:31:16Z)
• Preface: A Data-driven Volumetric Prior for Few-shot Ultra High-resolution Face Synthesis [0.0]
  NeRFs have enabled highly realistic synthesis of human faces including complex appearance and reflectance effects of hair and skin. We propose a novel human face prior that enables the synthesis of ultra high-resolution novel views of subjects that are not part of the prior's training distribution.
  arXiv  Detail & Related papers  (2023-09-28T21:21:44Z)
• Novel View Synthesis of Humans using Differentiable Rendering [50.57718384229912]
  We present a new approach for synthesizing novel views of people in new poses. Our synthesis makes use of diffuse Gaussian primitives that represent the underlying skeletal structure of a human. Rendering these primitives gives results in a high-dimensional latent image, which is then transformed into an RGB image by a decoder network.
  arXiv  Detail & Related papers  (2023-03-28T10:48:33Z)
• Human View Synthesis using a Single Sparse RGB-D Input [16.764379184593256]
  We present a novel view synthesis framework to generate realistic renders from unseen views of any human captured from a single-view sensor with sparse RGB-D. An enhancer network leverages the overall fidelity, even in occluded areas from the original view, producing crisp renders with fine details.
  arXiv  Detail & Related papers  (2021-12-27T20:13:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.