Generalizable Neural Human Renderer

• URL: http://arxiv.org/abs/2404.14199v1
• Date: Mon, 22 Apr 2024 14:09:53 GMT
• Title: Generalizable Neural Human Renderer
• Abstract summary: Generalizable Human Renderer (GNH) is a novel method for rendering animatable humans from monocular video without any test-time optimization. We demonstrate GNH surpasses current state-of-the-art methods, notably achieving a 31.3% improvement in LPIPS.
• Score: 29.533330467339013
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While recent advancements in animatable human rendering have achieved remarkable results, they require test-time optimization for each subject which can be a significant limitation for real-world applications. To address this, we tackle the challenging task of learning a Generalizable Neural Human Renderer (GNH), a novel method for rendering animatable humans from monocular video without any test-time optimization. Our core method focuses on transferring appearance information from the input video to the output image plane by utilizing explicit body priors and multi-view geometry. To render the subject in the intended pose, we utilize a straightforward CNN-based image renderer, foregoing the more common ray-sampling or rasterizing-based rendering modules. Our GNH achieves remarkable generalizable, photorealistic rendering with unseen subjects with a three-stage process. We quantitatively and qualitatively demonstrate that GNH significantly surpasses current state-of-the-art methods, notably achieving a 31.3% improvement in LPIPS.

Related papers

• Intrinsic Geometry-Appearance Consistency Optimization for Sparse-View Gaussian Splatting [36.3168821104293]
  3D human reconstruction from a single image is a challenging problem.<n>We present emphMVD-HuGaS, enabling free-view 3D human rendering from a single image via a multi-view human diffusion model.
  arXiv  Detail & Related papers  (2026-03-03T11:44:46Z)
• Parametric Gaussian Human Model: Generalizable Prior for Efficient and Realistic Human Avatar Modeling [32.480049588166544]
  Photo and animatable human avatars are a key enabler for virtual/augmented reality, telepresence, and digital entertainment.<n>We present the Parametric Gaussian Human Model (PGHM), a generalizable and efficient framework that integrates human priors into 3DGS.<n>Experiments show that PGHM is significantly more efficient than optimization-from-scratch methods, requiring only approximately 20 minutes per subject to produce avatars with comparable visual quality.
  arXiv  Detail & Related papers  (2025-06-07T03:53:30Z)
• GUAVA: Generalizable Upper Body 3D Gaussian Avatar [32.476282286315055]
  3D human avatar reconstruction typically requires multi-view or monocular videos and training on individual IDs.<n>We first introduce an expressive human model (EHM) to enhance facial expression capabilities.<n>We propose GUAVA, the first framework for fast animatable upper-body 3D Gaussian avatar reconstruction.
  arXiv  Detail & Related papers  (2025-05-06T09:19:16Z)
• MVD-HuGaS: Human Gaussians from a Single Image via 3D Human Multi-view Diffusion Prior [35.704591162502375]
  We present emphMVD-HuGaS, enabling free-view 3D human rendering from a single image via a multi-view human diffusion model. Experiments on Thuman2.0 and 2K2K datasets show that the proposed MVD-HuGaS achieves state-of-the-art performance on single-view 3D human rendering.
  arXiv  Detail & Related papers  (2025-03-11T09:37:15Z)
• HuGDiffusion: Generalizable Single-Image Human Rendering via 3D Gaussian Diffusion [50.02316409061741]
  HuGDiffusion is a learning pipeline to achieve novel view synthesis (NVS) of human characters from single-view input images. We aim to generate the set of 3DGS attributes via a diffusion-based framework conditioned on human priors extracted from a single image. Our HuGDiffusion shows significant performance improvements over the state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-01-25T01:00:33Z)
• iHuman: Instant Animatable Digital Humans From Monocular Videos [16.98924995658091]
  We present a fast, simple, yet effective method for creating animatable 3D digital humans from monocular videos. This work achieves and illustrates the need of accurate 3D mesh-type modelling of the human body. Our method is faster by an order of magnitude (in terms of training time) than its closest competitor.
  arXiv  Detail & Related papers  (2024-07-15T18:51:51Z)
• NPGA: Neural Parametric Gaussian Avatars [46.52887358194364]
  We propose a data-driven approach to create high-fidelity controllable avatars from multi-view video recordings. We build our method around 3D Gaussian splatting for its highly efficient rendering and to inherit the topological flexibility of point clouds. We evaluate our method on the public NeRSemble dataset, demonstrating that NPGA significantly outperforms the previous state-of-the-art avatars on the self-reenactment task by 2.6 PSNR.
  arXiv  Detail & Related papers  (2024-05-29T17:58:09Z)
• HR Human: Modeling Human Avatars with Triangular Mesh and High-Resolution Textures from Videos [52.23323966700072]
  We present a framework for acquiring human avatars that are attached with high-resolution physically-based material textures and mesh from monocular video. Our method introduces a novel information fusion strategy to combine the information from the monocular video and synthesize virtual multi-view images. Experiments show that our approach outperforms previous representations in terms of high fidelity, and this explicit result supports deployment on common triangulars.
  arXiv  Detail & Related papers  (2024-05-18T11:49:09Z)
• FaceFolds: Meshed Radiance Manifolds for Efficient Volumetric Rendering of Dynamic Faces [21.946327323788275]
  3D rendering of dynamic face is a challenging problem. We present a novel representation that enables high-quality rendering of an actor's dynamic facial performances.
  arXiv  Detail & Related papers  (2024-04-22T00:44:13Z)
• LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation [51.19871052619077]
  We introduce Large Multi-View Gaussian Model (LGM), a novel framework designed to generate high-resolution 3D models from text prompts or single-view images. We maintain the fast speed to generate 3D objects within 5 seconds while boosting the training resolution to 512, thereby achieving high-resolution 3D content generation.
  arXiv  Detail & Related papers  (2024-02-07T17:57:03Z)
• Differentiable Blocks World: Qualitative 3D Decomposition by Rendering Primitives [70.32817882783608]
  We present an approach that produces a simple, compact, and actionable 3D world representation by means of 3D primitives. Unlike existing primitive decomposition methods that rely on 3D input data, our approach operates directly on images. We show that the resulting textured primitives faithfully reconstruct the input images and accurately model the visible 3D points.
  arXiv  Detail & Related papers  (2023-07-11T17:58:31Z)
• Real-time volumetric rendering of dynamic humans [83.08068677139822]
  We present a method for fast 3D reconstruction and real-time rendering of dynamic humans from monocular videos. Our method can reconstruct a dynamic human in less than 3h using a single GPU, compared to recent state-of-the-art alternatives that take up to 72h. A novel local ray marching rendering allows visualizing the neural human on a mobile VR device at 40 frames per second with minimal loss of visual quality.
  arXiv  Detail & Related papers  (2023-03-21T14:41:25Z)
• High-fidelity 3D GAN Inversion by Pseudo-multi-view Optimization [51.878078860524795]
  We present a high-fidelity 3D generative adversarial network (GAN) inversion framework that can synthesize photo-realistic novel views. Our approach enables high-fidelity 3D rendering from a single image, which is promising for various applications of AI-generated 3D content.
  arXiv  Detail & Related papers  (2022-11-28T18:59:52Z)
• Inverting Generative Adversarial Renderer for Face Reconstruction [58.45125455811038]
  In this work, we introduce a novel Generative Adversa Renderer (GAR) GAR learns to model the complicated real-world image, instead of relying on the graphics rules, it is capable of producing realistic images. Our method achieves state-of-the-art performances on multiple face reconstruction.
  arXiv  Detail & Related papers  (2021-05-06T04:16:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.