HDhuman: High-quality Human Novel-view Rendering from Sparse Views
- URL: http://arxiv.org/abs/2201.08158v3
- Date: Sat, 21 Oct 2023 15:11:18 GMT
- Title: HDhuman: High-quality Human Novel-view Rendering from Sparse Views
- Authors: Tiansong Zhou, Jing Huang, Tao Yu, Ruizhi Shao, Kun Li
- Abstract summary: We propose HDhuman, which uses a human reconstruction network with a pixel-aligned spatial transformer and a rendering network with geometry-guided pixel-wise feature integration.
Our approach outperforms all the prior generic or specific methods on both synthetic data and real-world data.
- Score: 15.810495442598963
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: In this paper, we aim to address the challenge of novel view rendering of
human performers who wear clothes with complex texture patterns using a sparse
set of camera views. Although some recent works have achieved remarkable
rendering quality on humans with relatively uniform textures using sparse
views, the rendering quality remains limited when dealing with complex texture
patterns as they are unable to recover the high-frequency geometry details that
are observed in the input views. To this end, we propose HDhuman, which uses a
human reconstruction network with a pixel-aligned spatial transformer and a
rendering network with geometry-guided pixel-wise feature integration to
achieve high-quality human reconstruction and rendering. The designed
pixel-aligned spatial transformer calculates the correlations between the input
views and generates human reconstruction results with high-frequency details.
Based on the surface reconstruction results, the geometry-guided pixel-wise
visibility reasoning provides guidance for multi-view feature integration,
enabling the rendering network to render high-quality images at 2k resolution
on novel views. Unlike previous neural rendering works that always need to
train or fine-tune an independent network for a different scene, our method is
a general framework that is able to generalize to novel subjects. Experiments
show that our approach outperforms all the prior generic or specific methods on
both synthetic data and real-world data.
Related papers
- MaRINeR: Enhancing Novel Views by Matching Rendered Images with Nearby References [49.71130133080821]
MaRINeR is a refinement method that leverages information of a nearby mapping image to improve the rendering of a target viewpoint.
We show improved renderings in quantitative metrics and qualitative examples from both explicit and implicit scene representations.
arXiv Detail & Related papers (2024-07-18T17:50:03Z) - HR Human: Modeling Human Avatars with Triangular Mesh and High-Resolution Textures from Videos [52.23323966700072]
We present a framework for acquiring human avatars that are attached with high-resolution physically-based material textures and mesh from monocular video.
Our method introduces a novel information fusion strategy to combine the information from the monocular video and synthesize virtual multi-view images.
Experiments show that our approach outperforms previous representations in terms of high fidelity, and this explicit result supports deployment on common triangulars.
arXiv Detail & Related papers (2024-05-18T11:49:09Z) - GM-NeRF: Learning Generalizable Model-based Neural Radiance Fields from
Multi-view Images [79.39247661907397]
We introduce an effective framework Generalizable Model-based Neural Radiance Fields to synthesize free-viewpoint images.
Specifically, we propose a geometry-guided attention mechanism to register the appearance code from multi-view 2D images to a geometry proxy.
arXiv Detail & Related papers (2023-03-24T03:32:02Z) - Refining 3D Human Texture Estimation from a Single Image [3.8761064607384195]
Estimating 3D human texture from a single image is essential in graphics and vision.
We propose a framework that adaptively samples the input by a deformable convolution where offsets are learned via a deep neural network.
arXiv Detail & Related papers (2023-03-06T19:53:50Z) - High-fidelity 3D GAN Inversion by Pseudo-multi-view Optimization [51.878078860524795]
We present a high-fidelity 3D generative adversarial network (GAN) inversion framework that can synthesize photo-realistic novel views.
Our approach enables high-fidelity 3D rendering from a single image, which is promising for various applications of AI-generated 3D content.
arXiv Detail & Related papers (2022-11-28T18:59:52Z) - Human View Synthesis using a Single Sparse RGB-D Input [16.764379184593256]
We present a novel view synthesis framework to generate realistic renders from unseen views of any human captured from a single-view sensor with sparse RGB-D.
An enhancer network leverages the overall fidelity, even in occluded areas from the original view, producing crisp renders with fine details.
arXiv Detail & Related papers (2021-12-27T20:13:53Z) - NeuralHumanFVV: Real-Time Neural Volumetric Human Performance Rendering
using RGB Cameras [17.18904717379273]
4D reconstruction and rendering of human activities is critical for immersive VR/AR experience.
Recent advances still fail to recover fine geometry and texture results with the level of detail present in the input images from sparse multi-view RGB cameras.
We propose a real-time neural human performance capture and rendering system to generate both high-quality geometry and photo-realistic texture of human activities in arbitrary novel views.
arXiv Detail & Related papers (2021-03-13T12:03:38Z) - Neural Re-Rendering of Humans from a Single Image [80.53438609047896]
We propose a new method for neural re-rendering of a human under a novel user-defined pose and viewpoint.
Our algorithm represents body pose and shape as a parametric mesh which can be reconstructed from a single image.
arXiv Detail & Related papers (2021-01-11T18:53:47Z) - Pose-Guided High-Resolution Appearance Transfer via Progressive Training [65.92031716146865]
We propose a pose-guided appearance transfer network for transferring a given reference appearance to a target pose in unprecedented image resolution.
Our network utilizes dense local descriptors including local perceptual loss and local discriminators to refine details.
Our model produces high-quality images, which can be further utilized in useful applications such as garment transfer between people.
arXiv Detail & Related papers (2020-08-27T03:18:44Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.