HiFiHR: Enhancing 3D Hand Reconstruction from a Single Image via High-Fidelity Texture

• URL: http://arxiv.org/abs/2308.13628v1
• Date: Fri, 25 Aug 2023 18:48:40 GMT
• Title: HiFiHR: Enhancing 3D Hand Reconstruction from a Single Image via High-Fidelity Texture
• Authors: Jiayin Zhu, Zhuoran Zhao, Linlin Yang, Angela Yao
• Abstract summary: We present HiFiHR, a high-fidelity hand reconstruction approach that utilizes render-and-compare in the learning-based framework from a single image. Experimental results on public benchmarks including FreiHAND and HO-3D demonstrate that our method outperforms the state-of-the-art hand reconstruction methods in texture reconstruction quality.
• Score: 40.012406098563204
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present HiFiHR, a high-fidelity hand reconstruction approach that utilizes render-and-compare in the learning-based framework from a single image, capable of generating visually plausible and accurate 3D hand meshes while recovering realistic textures. Our method achieves superior texture reconstruction by employing a parametric hand model with predefined texture assets, and by establishing a texture reconstruction consistency between the rendered and input images during training. Moreover, based on pretraining the network on an annotated dataset, we apply varying degrees of supervision using our pipeline, i.e., self-supervision, weak supervision, and full supervision, and discuss the various levels of contributions of the learned high-fidelity textures in enhancing hand pose and shape estimation. Experimental results on public benchmarks including FreiHAND and HO-3D demonstrate that our method outperforms the state-of-the-art hand reconstruction methods in texture reconstruction quality while maintaining comparable accuracy in pose and shape estimation. Our code is available at https://github.com/viridityzhu/HiFiHR.

Related papers

• MaRINeR: Enhancing Novel Views by Matching Rendered Images with Nearby References [49.71130133080821]
  MaRINeR is a refinement method that leverages information of a nearby mapping image to improve the rendering of a target viewpoint. We show improved renderings in quantitative metrics and qualitative examples from both explicit and implicit scene representations.
  arXiv  Detail & Related papers  (2024-07-18T17:50:03Z)
• Fine-Grained Multi-View Hand Reconstruction Using Inverse Rendering [11.228453237603834]
  We present a novel fine-grained multi-view hand mesh reconstruction method that leverages inverse rendering to restore hand poses and intricate details. We also introduce a novel Hand Albedo and Mesh (HAM) optimization module to refine both the hand mesh and textures. Our proposed approach outperforms the state-of-the-art methods on both reconstruction accuracy and rendering quality.
  arXiv  Detail & Related papers  (2024-07-08T07:28:24Z)
• GTR: Improving Large 3D Reconstruction Models through Geometry and Texture Refinement [51.97726804507328]
  We propose a novel approach for 3D mesh reconstruction from multi-view images. Our method takes inspiration from large reconstruction models that use a transformer-based triplane generator and a Neural Radiance Field (NeRF) model trained on multi-view images.
  arXiv  Detail & Related papers  (2024-06-09T05:19:24Z)
• TMO: Textured Mesh Acquisition of Objects with a Mobile Device by using Differentiable Rendering [54.35405028643051]
  We present a new pipeline for acquiring a textured mesh in the wild with a single smartphone. Our method first introduces an RGBD-aided structure from motion, which can yield filtered depth maps. We adopt the neural implicit surface reconstruction method, which allows for high-quality mesh.
  arXiv  Detail & Related papers  (2023-03-27T10:07:52Z)
• High-fidelity 3D GAN Inversion by Pseudo-multi-view Optimization [51.878078860524795]
  We present a high-fidelity 3D generative adversarial network (GAN) inversion framework that can synthesize photo-realistic novel views. Our approach enables high-fidelity 3D rendering from a single image, which is promising for various applications of AI-generated 3D content.
  arXiv  Detail & Related papers  (2022-11-28T18:59:52Z)
• ReFu: Refine and Fuse the Unobserved View for Detail-Preserving Single-Image 3D Human Reconstruction [31.782985891629448]
  Single-image 3D human reconstruction aims to reconstruct the 3D textured surface of the human body given a single image. We propose ReFu, a coarse-to-fine approach that refines the projected backside view image and fuses the refined image to predict the final human body.
  arXiv  Detail & Related papers  (2022-11-09T09:14:11Z)
• Fast-GANFIT: Generative Adversarial Network for High Fidelity 3D Face Reconstruction [76.1612334630256]
  We harness the power of Generative Adversarial Networks (GANs) and Deep Convolutional Neural Networks (DCNNs) to reconstruct the facial texture and shape from single images. We demonstrate excellent results in photorealistic and identity preserving 3D face reconstructions and achieve for the first time, facial texture reconstruction with high-frequency details.
  arXiv  Detail & Related papers  (2021-05-16T16:35:44Z)
• Towards High-Fidelity 3D Face Reconstruction from In-the-Wild Images Using Graph Convolutional Networks [32.859340851346786]
  We introduce a method to reconstruct 3D facial shapes with high-fidelity textures from single-view images in-the-wild. Our method can generate high-quality results and outperforms state-of-the-art methods in both qualitative and quantitative comparisons.
  arXiv  Detail & Related papers  (2020-03-12T08:06:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.