LookinGood^{\pi}: Real-time Person-independent Neural Re-rendering for High-quality Human Performance Capture

• URL: http://arxiv.org/abs/2112.08037v1
• Date: Wed, 15 Dec 2021 11:00:21 GMT
• Title: LookinGood^{\pi}: Real-time Person-independent Neural Re-rendering for High-quality Human Performance Capture
• Authors: Xiqi Yang, Kewei Yang, Kang Chen, Weidong Zhang, Weiwei Xu
• Abstract summary: We propose a novel neural re-rendering approach that is aimed to improve the rendering quality of the low-quality reconstructed results from human performance capture system in real-time. Our key idea is to utilize the rendered image of reconstructed geometry as the guidance to assist the prediction of person-specific details from few reference images. We demonstrate that our method outperforms state-of-the-art methods at producing high-fidelity images on unseen people.
• Score: 13.026888802770902
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose LookinGood^{\pi}, a novel neural re-rendering approach that is aimed to (1) improve the rendering quality of the low-quality reconstructed results from human performance capture system in real-time; (2) improve the generalization ability of the neural rendering network on unseen people. Our key idea is to utilize the rendered image of reconstructed geometry as the guidance to assist the prediction of person-specific details from few reference images, thus enhancing the re-rendered result. In light of this, we design a two-branch network. A coarse branch is designed to fix some artifacts (i.e. holes, noise) and obtain a coarse version of the rendered input, while a detail branch is designed to predict "correct" details from the warped references. The guidance of the rendered image is realized by blending features from two branches effectively in the training of the detail branch, which improves both the warping accuracy and the details' fidelity. We demonstrate that our method outperforms state-of-the-art methods at producing high-fidelity images on unseen people.

Related papers

• MaRINeR: Enhancing Novel Views by Matching Rendered Images with Nearby References [49.71130133080821]
  MaRINeR is a refinement method that leverages information of a nearby mapping image to improve the rendering of a target viewpoint. We show improved renderings in quantitative metrics and qualitative examples from both explicit and implicit scene representations.
  arXiv  Detail & Related papers  (2024-07-18T17:50:03Z)
• ENTED: Enhanced Neural Texture Extraction and Distribution for Reference-based Blind Face Restoration [51.205673783866146]
  We present ENTED, a new framework for blind face restoration that aims to restore high-quality and realistic portrait images. We utilize a texture extraction and distribution framework to transfer high-quality texture features between the degraded input and reference image. The StyleGAN-like architecture in our framework requires high-quality latent codes to generate realistic images.
  arXiv  Detail & Related papers  (2024-01-13T04:54:59Z)
• InvertAvatar: Incremental GAN Inversion for Generalized Head Avatars [40.10906393484584]
  We propose a novel framework that enhances avatar reconstruction performance using an algorithm designed to increase the fidelity from multiple frames. Our architecture emphasizes pixel-aligned image-to-image translation, mitigating the need to learn correspondences between observation and canonical spaces. The proposed paradigm demonstrates state-of-the-art performance on one-shot and few-shot avatar animation tasks.
  arXiv  Detail & Related papers  (2023-12-03T18:59:15Z)
• Refining 3D Human Texture Estimation from a Single Image [3.8761064607384195]
  Estimating 3D human texture from a single image is essential in graphics and vision. We propose a framework that adaptively samples the input by a deformable convolution where offsets are learned via a deep neural network.
  arXiv  Detail & Related papers  (2023-03-06T19:53:50Z)
• FoVolNet: Fast Volume Rendering using Foveated Deep Neural Networks [33.489890950757975]
  FoVolNet is a method to significantly increase the performance of volume data visualization. We develop a cost-effective foveated rendering pipeline that sparsely samples a volume around a focal point and reconstructs the full-frame using a deep neural network.
  arXiv  Detail & Related papers  (2022-09-20T19:48:56Z)
• Towards Unsupervised Deep Image Enhancement with Generative Adversarial Network [92.01145655155374]
  We present an unsupervised image enhancement generative network (UEGAN) It learns the corresponding image-to-image mapping from a set of images with desired characteristics in an unsupervised manner. Results show that the proposed model effectively improves the aesthetic quality of images.
  arXiv  Detail & Related papers  (2020-12-30T03:22:46Z)
• Interpretable Detail-Fidelity Attention Network for Single Image Super-Resolution [89.1947690981471]
  We propose a purposeful and interpretable detail-fidelity attention network to progressively process smoothes and details in divide-and-conquer manner. Particularly, we propose a Hessian filtering for interpretable feature representation which is high-profile for detail inference. Experiments demonstrate that the proposed methods achieve superior performances over the state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-09-28T08:31:23Z)
• Towards Analysis-friendly Face Representation with Scalable Feature and Texture Compression [113.30411004622508]
  We show that a universal and collaborative visual information representation can be achieved in a hierarchical way. Based on the strong generative capability of deep neural networks, the gap between the base feature layer and enhancement layer is further filled with the feature level texture reconstruction. To improve the efficiency of the proposed framework, the base layer neural network is trained in a multi-task manner.
  arXiv  Detail & Related papers  (2020-04-21T14:32:49Z)
• Deep CG2Real: Synthetic-to-Real Translation via Image Disentanglement [78.58603635621591]
  Training an unpaired synthetic-to-real translation network in image space is severely under-constrained. We propose a semi-supervised approach that operates on the disentangled shading and albedo layers of the image. Our two-stage pipeline first learns to predict accurate shading in a supervised fashion using physically-based renderings as targets.
  arXiv  Detail & Related papers  (2020-03-27T21:45:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.